2022-11-23T01:10:29.4802960Z Requested labels: linux.8xlarge.nvidia.gpu
2022-11-23T01:10:29.4803062Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-11-23T01:10:29.4803099Z Waiting for a runner to pick up this job...
2022-11-23T01:10:29.6528132Z Job is about to start running on the runner: i-0a212d9242243d3a8 (organization)
2022-11-23T01:10:35.9722475Z Current runner version: '2.299.1'
2022-11-23T01:10:35.9730008Z Runner name: 'i-0a212d9242243d3a8'
2022-11-23T01:10:35.9730674Z Runner group name: 'Default'
2022-11-23T01:10:35.9731564Z Machine name: 'ip-10-0-3-252'
2022-11-23T01:10:35.9734273Z ##[group]GITHUB_TOKEN Permissions
2022-11-23T01:10:35.9735253Z Actions: write
2022-11-23T01:10:35.9735694Z Checks: write
2022-11-23T01:10:35.9736121Z Contents: write
2022-11-23T01:10:35.9736505Z Deployments: write
2022-11-23T01:10:35.9736940Z Discussions: write
2022-11-23T01:10:35.9737410Z Issues: write
2022-11-23T01:10:35.9737787Z Metadata: read
2022-11-23T01:10:35.9738603Z Packages: write
2022-11-23T01:10:35.9739047Z Pages: write
2022-11-23T01:10:35.9739431Z PullRequests: write
2022-11-23T01:10:35.9739963Z RepositoryProjects: write
2022-11-23T01:10:35.9740445Z SecurityEvents: write
2022-11-23T01:10:35.9740831Z Statuses: write
2022-11-23T01:10:35.9741262Z ##[endgroup]
2022-11-23T01:10:35.9745473Z Secret source: Actions
2022-11-23T01:10:35.9746288Z Prepare workflow directory
2022-11-23T01:10:36.1045267Z Prepare all required actions
2022-11-23T01:10:36.1267761Z Getting action download info
2022-11-23T01:10:36.3219434Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd)
2022-11-23T01:10:36.6328889Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f)
2022-11-23T01:10:39.7939855Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2022-11-23T01:10:40.0997117Z Getting action download info
2022-11-23T01:10:40.2482124Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-11-23T01:10:40.4155713Z Getting action download info
2022-11-23T01:10:40.6296002Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-11-23T01:10:40.7684045Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-11-23T01:10:40.7686459Z ##[group] Inputs
2022-11-23T01:10:40.7686864Z   build-environment: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:10:40.7688145Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "deploy", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
]}

2022-11-23T01:10:40.7689484Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:10:40.7689949Z   sync-tag: 
2022-11-23T01:10:40.7691000Z   timeout-minutes: 240
2022-11-23T01:10:40.7691287Z ##[endgroup]
2022-11-23T01:10:40.7692091Z Complete job name: linux-bionic-cuda11.6-py3.10-gcc7 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu)
2022-11-23T01:10:40.8754494Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2022-11-23T01:10:40.8754895Z with:
2022-11-23T01:10:40.8755464Z   github-secret: ***
2022-11-23T01:10:40.8755743Z   activate-with-label: false
2022-11-23T01:10:40.8756008Z   label: with-ssh
2022-11-23T01:10:40.8756269Z   remove-existing-keys: true
2022-11-23T01:10:40.8756510Z env:
2022-11-23T01:10:40.8756750Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:40.8757011Z ##[endgroup]
2022-11-23T01:10:40.9810652Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-11-23T01:10:41.0032909Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-11-23T01:10:41.0033285Z with:
2022-11-23T01:10:41.0033512Z   submodules: recursive
2022-11-23T01:10:41.0033767Z   fetch-depth: 0
2022-11-23T01:10:41.0033992Z env:
2022-11-23T01:10:41.0034231Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:41.0034531Z ##[endgroup]
2022-11-23T01:10:41.0322512Z ##[group]Run retry () {
2022-11-23T01:10:41.0322834Z retry () {
2022-11-23T01:10:41.0323127Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-11-23T01:10:41.0323418Z }
2022-11-23T01:10:41.0323675Z echo "${GITHUB_WORKSPACE}"
2022-11-23T01:10:41.0323954Z if [ -z "${NO_SUDO}" ]; then
2022-11-23T01:10:41.0324262Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:10:41.0324541Z else
2022-11-23T01:10:41.0324793Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:10:41.0325058Z fi
2022-11-23T01:10:41.0325351Z mkdir "${GITHUB_WORKSPACE}"
2022-11-23T01:10:41.0343572Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:10:41.0343878Z env:
2022-11-23T01:10:41.0344125Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:41.0344379Z   NO_SUDO: 
2022-11-23T01:10:41.0344598Z ##[endgroup]
2022-11-23T01:10:41.0469040Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:41.0913259Z ##[group]Run malfet/checkout@silent-checkout
2022-11-23T01:10:41.0913553Z with:
2022-11-23T01:10:41.0913819Z   ref: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:10:41.0914080Z   fetch-depth: 0
2022-11-23T01:10:41.0914333Z   submodules: recursive
2022-11-23T01:10:41.0914593Z   quiet-checkout: true
2022-11-23T01:10:41.0914845Z   repository: pytorch/pytorch
2022-11-23T01:10:41.0915294Z   token: ***
2022-11-23T01:10:41.0915700Z   ssh-strict: true
2022-11-23T01:10:41.0915975Z   persist-credentials: true
2022-11-23T01:10:41.0916220Z   clean: true
2022-11-23T01:10:41.0916454Z   lfs: false
2022-11-23T01:10:41.0916711Z   set-safe-directory: true
2022-11-23T01:10:41.0916942Z env:
2022-11-23T01:10:41.0917178Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:41.0917432Z ##[endgroup]
2022-11-23T01:10:41.2419134Z Syncing repository: pytorch/pytorch
2022-11-23T01:10:41.2420980Z ##[group]Getting Git version info
2022-11-23T01:10:41.2421526Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:10:41.2422105Z [command]/usr/bin/git version
2022-11-23T01:10:41.2422374Z git version 2.37.1
2022-11-23T01:10:41.2434553Z ##[endgroup]
2022-11-23T01:10:41.2454335Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/0cc579f2-81eb-4dae-b031-949c389cde0b' before making global git config changes
2022-11-23T01:10:41.2455748Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T01:10:41.2462297Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:41.2507180Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:10:41.2514219Z ##[group]Initializing the repository
2022-11-23T01:10:41.2518156Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:41.2549313Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-11-23T01:10:41.2549739Z hint: is subject to change. To configure the initial branch name to use in all
2022-11-23T01:10:41.2550196Z hint: of your new repositories, which will suppress this warning, call:
2022-11-23T01:10:41.2550512Z hint: 
2022-11-23T01:10:41.2550860Z hint: 	git config --global init.defaultBranch <name>
2022-11-23T01:10:41.2551147Z hint: 
2022-11-23T01:10:41.2551529Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-11-23T01:10:41.2551999Z hint: 'development'. The just-created branch can be renamed via this command:
2022-11-23T01:10:41.2552320Z hint: 
2022-11-23T01:10:41.2552756Z hint: 	git branch -m <name>
2022-11-23T01:10:41.2553355Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-11-23T01:10:41.2563420Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-11-23T01:10:41.2597336Z ##[endgroup]
2022-11-23T01:10:41.2597818Z ##[group]Disabling automatic garbage collection
2022-11-23T01:10:41.2602043Z [command]/usr/bin/git config --local gc.auto 0
2022-11-23T01:10:41.2633368Z ##[endgroup]
2022-11-23T01:10:41.2634382Z ##[group]Setting up auth
2022-11-23T01:10:41.2643669Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T01:10:41.2677153Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T01:10:41.3075125Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T01:10:41.3105464Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T01:10:41.3396489Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:10:41.3441356Z ##[endgroup]
2022-11-23T01:10:41.3441827Z ##[group]Fetching the repository
2022-11-23T01:10:41.3450559Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-11-23T01:11:31.6652707Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object}
2022-11-23T01:11:31.6679094Z 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:11:31.6686865Z ##[endgroup]
2022-11-23T01:11:31.6687374Z ##[group]Determining the checkout info
2022-11-23T01:11:31.6687832Z ##[endgroup]
2022-11-23T01:11:31.6688266Z ##[group]Checking out the ref
2022-11-23T01:11:31.6692664Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:11:33.3941476Z ##[endgroup]
2022-11-23T01:11:33.3942214Z ##[group]Setting up auth for fetching submodules
2022-11-23T01:11:33.3947938Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:11:33.4007391Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-11-23T01:11:33.4038174Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-11-23T01:11:33.4069206Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-11-23T01:11:33.4098345Z ##[endgroup]
2022-11-23T01:11:33.4099143Z ##[group]Fetching submodules
2022-11-23T01:11:33.4104221Z [command]/usr/bin/git submodule sync --recursive
2022-11-23T01:11:33.4420606Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-11-23T01:11:33.4719420Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-11-23T01:11:33.4721975Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-11-23T01:11:33.4724759Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-11-23T01:11:33.4727742Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-11-23T01:11:33.4731004Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-11-23T01:11:33.4734649Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-11-23T01:11:33.4738178Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-11-23T01:11:33.4742478Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-11-23T01:11:33.4746199Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-11-23T01:11:33.4750137Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-11-23T01:11:33.4754273Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-11-23T01:11:33.4758359Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-11-23T01:11:33.4762719Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-11-23T01:11:33.4767149Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-11-23T01:11:33.4771522Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-11-23T01:11:33.4776112Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-11-23T01:11:33.4781354Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-11-23T01:11:33.4786304Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:11:33.4791316Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-11-23T01:11:33.4796716Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-11-23T01:11:33.4802197Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-11-23T01:11:33.4807522Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-11-23T01:11:33.4813278Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-11-23T01:11:33.4819528Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-11-23T01:11:33.4825184Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-11-23T01:11:33.4831171Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-11-23T01:11:33.4837266Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-11-23T01:11:33.4843390Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-11-23T01:11:33.4849860Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-11-23T01:11:33.4856289Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-11-23T01:11:33.4863389Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-11-23T01:11:33.4870007Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-11-23T01:11:33.4876903Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-11-23T01:11:33.4883828Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-11-23T01:11:33.4891086Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-11-23T01:11:33.4898412Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-11-23T01:11:33.4905964Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-11-23T01:11:33.4913376Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-11-23T01:11:33.4921025Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-11-23T01:11:33.4928956Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-11-23T01:11:33.4936872Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-11-23T01:11:33.4965177Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-11-23T01:11:33.8293163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-11-23T01:11:34.0419873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-11-23T01:11:34.2700148Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-11-23T01:11:34.5785826Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-11-23T01:11:34.8568007Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-11-23T01:11:36.7233777Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-11-23T01:11:42.3124889Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-11-23T01:11:42.7025693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-11-23T01:11:43.2537685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-11-23T01:11:44.7540976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-11-23T01:11:46.0466297Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-11-23T01:11:47.7141824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-11-23T01:11:54.5437167Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-11-23T01:11:55.2818509Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-11-23T01:11:56.7492928Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-11-23T01:11:57.8294025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-11-23T01:11:58.0556787Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-11-23T01:11:58.5082167Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-11-23T01:11:58.8488118Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-11-23T01:11:59.8520004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-11-23T01:12:00.5393746Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-11-23T01:12:00.7442162Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-11-23T01:12:01.0496662Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-11-23T01:12:02.7566147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-11-23T01:12:03.2249895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-11-23T01:12:03.6267873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-11-23T01:12:09.7822807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-11-23T01:12:11.5143046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-11-23T01:12:11.9480758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-11-23T01:12:12.2039641Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-11-23T01:12:18.1797062Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-11-23T01:12:18.3779954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-11-23T01:12:18.6099496Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-11-23T01:12:19.4677624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-11-23T01:12:19.6916435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-11-23T01:12:20.0894552Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-11-23T01:12:20.5259630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-11-23T01:12:21.1214679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-11-23T01:12:23.5666581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-11-23T01:12:24.0899385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-11-23T01:12:26.2689772Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-11-23T01:12:26.2813601Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-11-23T01:12:26.2907058Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-11-23T01:12:26.3187115Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-11-23T01:12:26.3461654Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-11-23T01:12:26.3881394Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-11-23T01:12:27.1408600Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-11-23T01:12:27.1658273Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:12:27.2869708Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-11-23T01:12:27.3262283Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-11-23T01:12:27.6825317Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-11-23T01:12:28.2020055Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-11-23T01:12:28.5010965Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-11-23T01:12:28.5553217Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a'
2022-11-23T01:12:28.5570538Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:12:28.5573677Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:12:28.5576828Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:12:28.5580919Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:12:28.5607858Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-11-23T01:12:29.4878948Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-11-23T01:12:30.0402250Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-11-23T01:12:31.0421171Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-11-23T01:12:31.3655397Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-11-23T01:12:31.4880712Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-11-23T01:12:31.5563027Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-11-23T01:12:31.5672928Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-11-23T01:12:31.6826157Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-11-23T01:12:31.7244785Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2022-11-23T01:12:31.7342982Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-11-23T01:12:31.7802952Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-11-23T01:12:31.8082820Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2022-11-23T01:12:31.8623130Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-11-23T01:12:31.8750978Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f'
2022-11-23T01:12:31.8767236Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-11-23T01:12:31.8793298Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-11-23T01:12:40.3339307Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d'
2022-11-23T01:12:40.3359664Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:12:40.3387152Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-11-23T01:12:48.9724301Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289'
2022-11-23T01:12:48.9839329Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-11-23T01:12:49.0004255Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-11-23T01:12:49.1110028Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2022-11-23T01:12:49.1127873Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:12:49.1131162Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:12:49.1158953Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-11-23T01:12:50.1958034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-11-23T01:12:51.2303692Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-11-23T01:12:51.2940447Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-11-23T01:12:51.3175835Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-11-23T01:12:51.3328522Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-11-23T01:12:51.4613761Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-11-23T01:12:51.7675510Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-11-23T01:12:51.7706759Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-11-23T01:12:51.7710094Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-11-23T01:12:51.7736815Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-11-23T01:12:52.2115158Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-11-23T01:12:53.0951415Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:12:53.1328757Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-11-23T01:12:53.1503447Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-11-23T01:12:53.1520487Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:12:53.1546447Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-11-23T01:12:54.8697688Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-11-23T01:12:54.8720233Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:12:54.8723581Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:12:54.8750748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-11-23T01:12:55.2771729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-11-23T01:12:56.1281862Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-11-23T01:12:56.2053435Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-11-23T01:12:56.2069530Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:12:56.2095452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-11-23T01:12:56.4202914Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:12:56.4305025Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-11-23T01:12:56.7442713Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-11-23T01:12:56.7465155Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:12:56.7468929Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-11-23T01:12:56.7525126Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-11-23T01:12:57.1757216Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-11-23T01:12:59.5593750Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-11-23T01:12:59.6410693Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-11-23T01:12:59.6502257Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-11-23T01:12:59.6629646Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-11-23T01:12:59.7022965Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2022-11-23T01:12:59.7122782Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-11-23T01:12:59.7451260Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-11-23T01:12:59.7555609Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-11-23T01:12:59.8075274Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-11-23T01:12:59.9486275Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-11-23T01:12:59.9797384Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-11-23T01:12:59.9815169Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:12:59.9818291Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:12:59.9822050Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:12:59.9825300Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:12:59.9851759Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-11-23T01:13:00.9898324Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-11-23T01:13:01.3673056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-11-23T01:13:02.8271626Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-11-23T01:13:03.8299484Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-11-23T01:13:03.8465136Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-11-23T01:13:03.9239369Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-11-23T01:13:03.9565109Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-11-23T01:13:03.9582697Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:03.9607925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-11-23T01:13:04.2154292Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:13:04.3729660Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-11-23T01:13:04.3769950Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-11-23T01:13:04.4086419Z Entering 'android/libs/fbjni'
2022-11-23T01:13:04.4129247Z Entering 'third_party/FP16'
2022-11-23T01:13:04.4171935Z Entering 'third_party/FXdiv'
2022-11-23T01:13:04.4214291Z Entering 'third_party/NNPACK'
2022-11-23T01:13:04.4257110Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:04.4301319Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:04.4344676Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:04.4397849Z Entering 'third_party/benchmark'
2022-11-23T01:13:04.4439608Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:04.4482086Z Entering 'third_party/cub'
2022-11-23T01:13:04.4524100Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:04.4572214Z Entering 'third_party/cutlass'
2022-11-23T01:13:04.4621464Z Entering 'third_party/eigen'
2022-11-23T01:13:04.4666317Z Entering 'third_party/fbgemm'
2022-11-23T01:13:04.4708682Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:04.4750898Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:04.4793947Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:04.4834896Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:04.4877799Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:04.4921464Z Entering 'third_party/fmt'
2022-11-23T01:13:04.4962611Z Entering 'third_party/foxi'
2022-11-23T01:13:04.5005492Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:04.5046458Z Entering 'third_party/gloo'
2022-11-23T01:13:04.5087949Z Entering 'third_party/googletest'
2022-11-23T01:13:04.5129932Z Entering 'third_party/ideep'
2022-11-23T01:13:04.5171183Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:04.5215747Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:04.5264331Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:04.5305413Z Entering 'third_party/ittapi'
2022-11-23T01:13:04.5347155Z Entering 'third_party/kineto'
2022-11-23T01:13:04.5388707Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:04.5430642Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:04.5472837Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:04.5513611Z Entering 'third_party/neon2sse'
2022-11-23T01:13:04.5554768Z Entering 'third_party/nlohmann'
2022-11-23T01:13:04.5597719Z Entering 'third_party/onnx'
2022-11-23T01:13:04.5651699Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:04.5693252Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:04.5736281Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:04.5776519Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:04.5824021Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:04.5865243Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:04.5906141Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:04.5952240Z Entering 'third_party/pocketfft'
2022-11-23T01:13:04.5992804Z Entering 'third_party/protobuf'
2022-11-23T01:13:04.6038221Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:04.6078594Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:04.6122500Z Entering 'third_party/psimd'
2022-11-23T01:13:04.6163413Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:04.6204775Z Entering 'third_party/pybind11'
2022-11-23T01:13:04.6245933Z Entering 'third_party/python-enum'
2022-11-23T01:13:04.6287272Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:04.6329369Z Entering 'third_party/python-six'
2022-11-23T01:13:04.6370627Z Entering 'third_party/sleef'
2022-11-23T01:13:04.6411950Z Entering 'third_party/tbb'
2022-11-23T01:13:04.6455868Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:04.6498409Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:04.6539104Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:04.6580181Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:04.6622076Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:04.6663992Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:04.6708114Z Entering 'third_party/zstd'
2022-11-23T01:13:04.6762053Z ##[endgroup]
2022-11-23T01:13:04.6762673Z ##[group]Persisting credentials for submodules
2022-11-23T01:13:04.6768420Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-11-23T01:13:04.7081760Z Entering 'android/libs/fbjni'
2022-11-23T01:13:04.7123498Z Entering 'third_party/FP16'
2022-11-23T01:13:04.7165771Z Entering 'third_party/FXdiv'
2022-11-23T01:13:04.7208223Z Entering 'third_party/NNPACK'
2022-11-23T01:13:04.7250775Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:04.7293037Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:04.7336526Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:04.7389698Z Entering 'third_party/benchmark'
2022-11-23T01:13:04.7431584Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:04.7477155Z Entering 'third_party/cub'
2022-11-23T01:13:04.7519613Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:04.7568143Z Entering 'third_party/cutlass'
2022-11-23T01:13:04.7617993Z Entering 'third_party/eigen'
2022-11-23T01:13:04.7662826Z Entering 'third_party/fbgemm'
2022-11-23T01:13:04.7706049Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:04.7748586Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:04.7790719Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:04.7831841Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:04.7874002Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:04.7918223Z Entering 'third_party/fmt'
2022-11-23T01:13:04.7963806Z Entering 'third_party/foxi'
2022-11-23T01:13:04.8005043Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:04.8047138Z Entering 'third_party/gloo'
2022-11-23T01:13:04.8088714Z Entering 'third_party/googletest'
2022-11-23T01:13:04.8130380Z Entering 'third_party/ideep'
2022-11-23T01:13:04.8172405Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:04.8214714Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:04.8263166Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:04.8304313Z Entering 'third_party/ittapi'
2022-11-23T01:13:04.8346118Z Entering 'third_party/kineto'
2022-11-23T01:13:04.8386911Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:04.8428137Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:04.8469424Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:04.8511293Z Entering 'third_party/neon2sse'
2022-11-23T01:13:04.8551904Z Entering 'third_party/nlohmann'
2022-11-23T01:13:04.8595068Z Entering 'third_party/onnx'
2022-11-23T01:13:04.8650581Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:04.8691569Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:04.8735379Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:04.8775457Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:04.8821881Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:04.8862815Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:04.8904541Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:04.8950016Z Entering 'third_party/pocketfft'
2022-11-23T01:13:04.8990511Z Entering 'third_party/protobuf'
2022-11-23T01:13:04.9034268Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:04.9075133Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:04.9118128Z Entering 'third_party/psimd'
2022-11-23T01:13:04.9159084Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:04.9200571Z Entering 'third_party/pybind11'
2022-11-23T01:13:04.9243288Z Entering 'third_party/python-enum'
2022-11-23T01:13:04.9284935Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:04.9325999Z Entering 'third_party/python-six'
2022-11-23T01:13:04.9367542Z Entering 'third_party/sleef'
2022-11-23T01:13:04.9409121Z Entering 'third_party/tbb'
2022-11-23T01:13:04.9452763Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:04.9494328Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:04.9535324Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:04.9576369Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:04.9616827Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:04.9656512Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:04.9700148Z Entering 'third_party/zstd'
2022-11-23T01:13:04.9755275Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-11-23T01:13:05.0062786Z Entering 'android/libs/fbjni'
2022-11-23T01:13:05.0100132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-11-23T01:13:05.0117084Z Entering 'third_party/FP16'
2022-11-23T01:13:05.0155566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-11-23T01:13:05.0172272Z Entering 'third_party/FXdiv'
2022-11-23T01:13:05.0211733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-11-23T01:13:05.0228566Z Entering 'third_party/NNPACK'
2022-11-23T01:13:05.0266929Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-11-23T01:13:05.0283667Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:05.0322079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-11-23T01:13:05.0339256Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:05.0377632Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-11-23T01:13:05.0395041Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:05.0433715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-11-23T01:13:05.0462695Z Entering 'third_party/benchmark'
2022-11-23T01:13:05.0500703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:05.0517504Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:05.0555812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:13:05.0573041Z Entering 'third_party/cub'
2022-11-23T01:13:05.0610959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-11-23T01:13:05.0628792Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:05.0666395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-11-23T01:13:05.0689311Z Entering 'third_party/cutlass'
2022-11-23T01:13:05.0727022Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-11-23T01:13:05.0751462Z Entering 'third_party/eigen'
2022-11-23T01:13:05.0789245Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-11-23T01:13:05.0808569Z Entering 'third_party/fbgemm'
2022-11-23T01:13:05.0847358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-11-23T01:13:05.0864909Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:05.0902181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-11-23T01:13:05.0919008Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:05.0957087Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:13:05.0974571Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:05.1011449Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.1029263Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:05.1066870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-11-23T01:13:05.1085241Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:05.1122770Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-11-23T01:13:05.1141978Z Entering 'third_party/fmt'
2022-11-23T01:13:05.1178598Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-11-23T01:13:05.1195936Z Entering 'third_party/foxi'
2022-11-23T01:13:05.1233848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-11-23T01:13:05.1250306Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:05.1288452Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-11-23T01:13:05.1306301Z Entering 'third_party/gloo'
2022-11-23T01:13:05.1344181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-11-23T01:13:05.1361086Z Entering 'third_party/googletest'
2022-11-23T01:13:05.1399006Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.1416260Z Entering 'third_party/ideep'
2022-11-23T01:13:05.1454504Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-11-23T01:13:05.1472663Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:05.1510899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-11-23T01:13:05.1529358Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:05.1566941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-11-23T01:13:05.1591239Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:05.1628679Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-11-23T01:13:05.1645508Z Entering 'third_party/ittapi'
2022-11-23T01:13:05.1683632Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-11-23T01:13:05.1701049Z Entering 'third_party/kineto'
2022-11-23T01:13:05.1738420Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-11-23T01:13:05.1755984Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:05.1793440Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-11-23T01:13:05.1810629Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:05.1849428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.1868492Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:05.1906262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-11-23T01:13:05.1923439Z Entering 'third_party/neon2sse'
2022-11-23T01:13:05.1961041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-11-23T01:13:05.1977829Z Entering 'third_party/nlohmann'
2022-11-23T01:13:05.2015413Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-11-23T01:13:05.2034190Z Entering 'third_party/onnx'
2022-11-23T01:13:05.2072111Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:13:05.2102129Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.2139618Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:05.2157992Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.2196083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:05.2215434Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:05.2252698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-11-23T01:13:05.2270302Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:05.2308256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:13:05.2330072Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.2368419Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:05.2386082Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.2424187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:05.2440711Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.2479969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:13:05.2501846Z Entering 'third_party/pocketfft'
2022-11-23T01:13:05.2541017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-11-23T01:13:05.2557907Z Entering 'third_party/protobuf'
2022-11-23T01:13:05.2596367Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-11-23T01:13:05.2616486Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:05.2654070Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:05.2671193Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:05.2708967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.2727617Z Entering 'third_party/psimd'
2022-11-23T01:13:05.2765375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-11-23T01:13:05.2782977Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:05.2821505Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-11-23T01:13:05.2838352Z Entering 'third_party/pybind11'
2022-11-23T01:13:05.2876826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:05.2894084Z Entering 'third_party/python-enum'
2022-11-23T01:13:05.2932325Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-11-23T01:13:05.2950043Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:05.2988238Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-11-23T01:13:05.3005506Z Entering 'third_party/python-six'
2022-11-23T01:13:05.3044286Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-11-23T01:13:05.3061652Z Entering 'third_party/sleef'
2022-11-23T01:13:05.3098964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-11-23T01:13:05.3117896Z Entering 'third_party/tbb'
2022-11-23T01:13:05.3155860Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-11-23T01:13:05.3174952Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:05.3215015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-11-23T01:13:05.3232906Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:05.3270088Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.3286841Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:05.3324708Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-11-23T01:13:05.3341856Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:05.3378902Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-11-23T01:13:05.3396236Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:05.3433896Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:05.3450359Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.3488923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:13:05.3509587Z Entering 'third_party/zstd'
2022-11-23T01:13:05.3547154Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-11-23T01:13:05.4574947Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-11-23T01:13:05.4885774Z Entering 'android/libs/fbjni'
2022-11-23T01:13:05.4928452Z Entering 'third_party/FP16'
2022-11-23T01:13:05.4970831Z Entering 'third_party/FXdiv'
2022-11-23T01:13:05.5013373Z Entering 'third_party/NNPACK'
2022-11-23T01:13:05.5056296Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:05.5099282Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:05.5142696Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:05.5195649Z Entering 'third_party/benchmark'
2022-11-23T01:13:05.5237853Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:05.5280450Z Entering 'third_party/cub'
2022-11-23T01:13:05.5322119Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:05.5369823Z Entering 'third_party/cutlass'
2022-11-23T01:13:05.5418830Z Entering 'third_party/eigen'
2022-11-23T01:13:05.5463543Z Entering 'third_party/fbgemm'
2022-11-23T01:13:05.5505688Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:05.5547031Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:05.5588575Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:05.5630594Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:05.5672263Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:05.5716188Z Entering 'third_party/fmt'
2022-11-23T01:13:05.5757916Z Entering 'third_party/foxi'
2022-11-23T01:13:05.5800267Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:05.5842525Z Entering 'third_party/gloo'
2022-11-23T01:13:05.5884748Z Entering 'third_party/googletest'
2022-11-23T01:13:05.5926673Z Entering 'third_party/ideep'
2022-11-23T01:13:05.5967699Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:05.6013279Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:05.6063503Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:05.6105507Z Entering 'third_party/ittapi'
2022-11-23T01:13:05.6146688Z Entering 'third_party/kineto'
2022-11-23T01:13:05.6189065Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:05.6230747Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:05.6273243Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:05.6316091Z Entering 'third_party/neon2sse'
2022-11-23T01:13:05.6358006Z Entering 'third_party/nlohmann'
2022-11-23T01:13:05.6400405Z Entering 'third_party/onnx'
2022-11-23T01:13:05.6455451Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.6497349Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.6541402Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:05.6583298Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:05.6630364Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.6671966Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.6713657Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.6759790Z Entering 'third_party/pocketfft'
2022-11-23T01:13:05.6802593Z Entering 'third_party/protobuf'
2022-11-23T01:13:05.6849773Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:05.6891881Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:05.6935252Z Entering 'third_party/psimd'
2022-11-23T01:13:05.6977332Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:05.7019708Z Entering 'third_party/pybind11'
2022-11-23T01:13:05.7062160Z Entering 'third_party/python-enum'
2022-11-23T01:13:05.7104313Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:05.7146191Z Entering 'third_party/python-six'
2022-11-23T01:13:05.7187819Z Entering 'third_party/sleef'
2022-11-23T01:13:05.7229417Z Entering 'third_party/tbb'
2022-11-23T01:13:05.7272894Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:05.7315733Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:05.7357133Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:05.7398271Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:05.7440006Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:05.7480825Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.7525624Z Entering 'third_party/zstd'
2022-11-23T01:13:05.7580705Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-11-23T01:13:05.7885990Z Entering 'android/libs/fbjni'
2022-11-23T01:13:05.7927829Z Entering 'third_party/FP16'
2022-11-23T01:13:05.7970050Z Entering 'third_party/FXdiv'
2022-11-23T01:13:05.8011837Z Entering 'third_party/NNPACK'
2022-11-23T01:13:05.8054230Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:05.8096493Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:05.8139840Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:05.8191804Z Entering 'third_party/benchmark'
2022-11-23T01:13:05.8234140Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:05.8276931Z Entering 'third_party/cub'
2022-11-23T01:13:05.8319479Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:05.8367470Z Entering 'third_party/cutlass'
2022-11-23T01:13:05.8416312Z Entering 'third_party/eigen'
2022-11-23T01:13:05.8460882Z Entering 'third_party/fbgemm'
2022-11-23T01:13:05.8503492Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:05.8546007Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:05.8588837Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:05.8631056Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:05.8674194Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:05.8717565Z Entering 'third_party/fmt'
2022-11-23T01:13:05.8758982Z Entering 'third_party/foxi'
2022-11-23T01:13:05.8801225Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:05.8842531Z Entering 'third_party/gloo'
2022-11-23T01:13:05.8884623Z Entering 'third_party/googletest'
2022-11-23T01:13:05.8926950Z Entering 'third_party/ideep'
2022-11-23T01:13:05.8967514Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:05.9011359Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:05.9059516Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:05.9101948Z Entering 'third_party/ittapi'
2022-11-23T01:13:05.9143743Z Entering 'third_party/kineto'
2022-11-23T01:13:05.9185550Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:05.9226988Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:05.9269475Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:05.9311473Z Entering 'third_party/neon2sse'
2022-11-23T01:13:05.9352371Z Entering 'third_party/nlohmann'
2022-11-23T01:13:05.9394749Z Entering 'third_party/onnx'
2022-11-23T01:13:05.9450002Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.9491677Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.9535610Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:05.9576397Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:05.9623401Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.9666027Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.9707374Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.9753609Z Entering 'third_party/pocketfft'
2022-11-23T01:13:05.9795738Z Entering 'third_party/protobuf'
2022-11-23T01:13:05.9840800Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:05.9882042Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:05.9925904Z Entering 'third_party/psimd'
2022-11-23T01:13:05.9968088Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:06.0009456Z Entering 'third_party/pybind11'
2022-11-23T01:13:06.0051761Z Entering 'third_party/python-enum'
2022-11-23T01:13:06.0093172Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:06.0135869Z Entering 'third_party/python-six'
2022-11-23T01:13:06.0177591Z Entering 'third_party/sleef'
2022-11-23T01:13:06.0222794Z Entering 'third_party/tbb'
2022-11-23T01:13:06.0266669Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:06.0309301Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:06.0352080Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:06.0394605Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:06.0436447Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:06.0477501Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:06.0522190Z Entering 'third_party/zstd'
2022-11-23T01:13:06.0573207Z ##[endgroup]
2022-11-23T01:13:06.0624421Z [command]/usr/bin/git log -1 --format='%H'
2022-11-23T01:13:06.0652355Z '1cfd3858ac54fe3883534309081631a0a892ba3f'
2022-11-23T01:13:06.0803209Z Prepare all required actions
2022-11-23T01:13:06.0834743Z ##[group]Run ./.github/actions/setup-linux
2022-11-23T01:13:06.0835023Z env:
2022-11-23T01:13:06.0835266Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:06.0835504Z ##[endgroup]
2022-11-23T01:13:06.0852830Z ##[group]Run set -euo pipefail
2022-11-23T01:13:06.0853148Z set -euo pipefail
2022-11-23T01:13:06.0853436Z function get_ec2_metadata() {
2022-11-23T01:13:06.0853773Z   # Pulled from instance metadata endpoint for EC2
2022-11-23T01:13:06.0854230Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-11-23T01:13:06.0854624Z   category=$1
2022-11-23T01:13:06.0854950Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-11-23T01:13:06.0855238Z }
2022-11-23T01:13:06.0855510Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-11-23T01:13:06.0855890Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-11-23T01:13:06.0856248Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-11-23T01:13:06.0856590Z echo "system info $(uname -a)"
2022-11-23T01:13:06.0869880Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:06.0870158Z env:
2022-11-23T01:13:06.0870399Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:06.0870658Z ##[endgroup]
2022-11-23T01:13:06.0970741Z ami-id: ami-096198a0bccc6bad4
2022-11-23T01:13:06.1033655Z instance-id: i-0a212d9242243d3a8
2022-11-23T01:13:06.1094944Z instance-type: g3.8xlarge
2022-11-23T01:13:06.1103524Z system info Linux ip-10-0-3-252.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-11-23T01:13:06.1122274Z ##[group]Run if systemctl is-active --quiet docker; then
2022-11-23T01:13:06.1122662Z if systemctl is-active --quiet docker; then
2022-11-23T01:13:06.1122982Z     echo "Docker daemon is running...";
2022-11-23T01:13:06.1123272Z else
2022-11-23T01:13:06.1123591Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-11-23T01:13:06.1123894Z fi
2022-11-23T01:13:06.1135428Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:06.1135727Z env:
2022-11-23T01:13:06.1135950Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:06.1136210Z ##[endgroup]
2022-11-23T01:13:06.1188693Z Docker daemon is running...
2022-11-23T01:13:06.1206663Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:13:06.1207181Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:13:06.1207564Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:06.1208048Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-11-23T01:13:06.1208500Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-11-23T01:13:06.1220458Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:06.1220764Z env:
2022-11-23T01:13:06.1220987Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:06.1221257Z   AWS_RETRY_MODE: standard
2022-11-23T01:13:06.1221516Z   AWS_MAX_ATTEMPTS: 5
2022-11-23T01:13:06.1221768Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T01:13:06.1222030Z ##[endgroup]
2022-11-23T01:13:07.0492691Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-11-23T01:13:07.0493151Z Configure a credential helper to remove this warning. See
2022-11-23T01:13:07.0494790Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-11-23T01:13:07.0495097Z 
2022-11-23T01:13:07.0495194Z Login Succeeded
2022-11-23T01:13:07.0565479Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:07.0566173Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:07.0566690Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:07.0579820Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:07.0580113Z env:
2022-11-23T01:13:07.0580356Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:07.0580602Z ##[endgroup]
2022-11-23T01:13:07.0667069Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2022-11-23T01:13:07.0667408Z with:
2022-11-23T01:13:07.0667888Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:13:07.0668350Z env:
2022-11-23T01:13:07.0668588Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:07.0668831Z ##[endgroup]
2022-11-23T01:13:07.0683856Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:07.0684221Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:07.0684579Z # ignore output since only exit code is used for conditional
2022-11-23T01:13:07.0684960Z # only pull docker image if it's not available locally
2022-11-23T01:13:07.0685348Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-11-23T01:13:07.0685741Z   retry docker pull "${DOCKER_IMAGE}"
2022-11-23T01:13:07.0686017Z fi
2022-11-23T01:13:07.0697528Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:07.0697804Z env:
2022-11-23T01:13:07.0698415Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:07.0698932Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:13:07.0699408Z ##[endgroup]
2022-11-23T01:13:07.3611919Z 072aae4a77ed7d3a69ad5683420509c41301b940: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7
2022-11-23T01:13:07.3612808Z a404e5416296: Pulling fs layer
2022-11-23T01:13:07.3613313Z c58c079e9b17: Pulling fs layer
2022-11-23T01:13:07.3613820Z e5b80b8bbe91: Pulling fs layer
2022-11-23T01:13:07.3614354Z 888240790290: Pulling fs layer
2022-11-23T01:13:07.3614868Z 515fe5e34eb4: Pulling fs layer
2022-11-23T01:13:07.3615390Z 4e4521f12f5a: Pulling fs layer
2022-11-23T01:13:07.3615927Z f6e1a56cb32d: Pulling fs layer
2022-11-23T01:13:07.3616438Z c29b96e36bd0: Pulling fs layer
2022-11-23T01:13:07.3616934Z 304d3c6c28d0: Pulling fs layer
2022-11-23T01:13:07.3617436Z fac00e927cfe: Pulling fs layer
2022-11-23T01:13:07.3617851Z f0158c8d8420: Pulling fs layer
2022-11-23T01:13:07.3618542Z 3ceac802dd07: Pulling fs layer
2022-11-23T01:13:07.3618805Z 0d0e625ba887: Pulling fs layer
2022-11-23T01:13:07.3619077Z bc2be817cb7e: Pulling fs layer
2022-11-23T01:13:07.3619353Z 11eb2106b948: Pulling fs layer
2022-11-23T01:13:07.3619603Z 34fa4193c7a6: Pulling fs layer
2022-11-23T01:13:07.3619872Z a7cf5b3894f8: Pulling fs layer
2022-11-23T01:13:07.3620125Z 888240790290: Waiting
2022-11-23T01:13:07.3620389Z 3f6b06edd3f5: Pulling fs layer
2022-11-23T01:13:07.3620646Z 73a2b1f75a3d: Pulling fs layer
2022-11-23T01:13:07.3623094Z ba6235196410: Pulling fs layer
2022-11-23T01:13:07.3623506Z 879cdaf83543: Pulling fs layer
2022-11-23T01:13:07.3623779Z 3ceac802dd07: Waiting
2022-11-23T01:13:07.3624042Z fac00e927cfe: Waiting
2022-11-23T01:13:07.3624301Z 6ff0fc00b0a9: Pulling fs layer
2022-11-23T01:13:07.3624584Z a58b9ed071f4: Pulling fs layer
2022-11-23T01:13:07.3624851Z 515fe5e34eb4: Waiting
2022-11-23T01:13:07.3625078Z 4e4521f12f5a: Waiting
2022-11-23T01:13:07.3625320Z 304d3c6c28d0: Waiting
2022-11-23T01:13:07.3631889Z f6e1a56cb32d: Waiting
2022-11-23T01:13:07.3632247Z a8c562f6a1cf: Pulling fs layer
2022-11-23T01:13:07.3632544Z 0a39b4492650: Pulling fs layer
2022-11-23T01:13:07.3632818Z 9088ff8de269: Pulling fs layer
2022-11-23T01:13:07.3633064Z 34fa4193c7a6: Waiting
2022-11-23T01:13:07.3633323Z 165006759af3: Pulling fs layer
2022-11-23T01:13:07.3633581Z 3f6b06edd3f5: Waiting
2022-11-23T01:13:07.3633806Z ba6235196410: Waiting
2022-11-23T01:13:07.3634057Z ae48b7377a0d: Pulling fs layer
2022-11-23T01:13:07.3634312Z 73a2b1f75a3d: Waiting
2022-11-23T01:13:07.3634740Z b18965f4b6f1: Pulling fs layer
2022-11-23T01:13:07.3634997Z 879cdaf83543: Waiting
2022-11-23T01:13:07.3635239Z a8c562f6a1cf: Waiting
2022-11-23T01:13:07.3635458Z 165006759af3: Waiting
2022-11-23T01:13:07.3635812Z 102ddcd90753: Pulling fs layer
2022-11-23T01:13:07.3636084Z 0a39b4492650: Waiting
2022-11-23T01:13:07.3636305Z 9088ff8de269: Waiting
2022-11-23T01:13:07.3636561Z 5f5dd1cba120: Pulling fs layer
2022-11-23T01:13:07.3636823Z b18965f4b6f1: Waiting
2022-11-23T01:13:07.3637065Z 8a7f50c8b503: Pulling fs layer
2022-11-23T01:13:07.3637329Z bc2be817cb7e: Waiting
2022-11-23T01:13:07.3637585Z 863c35620b44: Pulling fs layer
2022-11-23T01:13:07.3637816Z 0d0e625ba887: Waiting
2022-11-23T01:13:07.3638068Z 183e4209dc37: Pulling fs layer
2022-11-23T01:13:07.3638319Z a7cf5b3894f8: Waiting
2022-11-23T01:13:07.3638576Z a47cba6c334e: Pulling fs layer
2022-11-23T01:13:07.3638844Z 102ddcd90753: Waiting
2022-11-23T01:13:07.3639093Z a9f3d4742233: Pulling fs layer
2022-11-23T01:13:07.3639364Z 3cefa8a4607f: Pulling fs layer
2022-11-23T01:13:07.3639628Z 023a41fa48e6: Pulling fs layer
2022-11-23T01:13:07.3639879Z 183e4209dc37: Waiting
2022-11-23T01:13:07.3640129Z 96e251412f4d: Pulling fs layer
2022-11-23T01:13:07.3640373Z c29b96e36bd0: Waiting
2022-11-23T01:13:07.3640846Z 49d40c00cf56: Pulling fs layer
2022-11-23T01:13:07.3641109Z 7e2d6313145f: Pulling fs layer
2022-11-23T01:13:07.3641340Z 11eb2106b948: Waiting
2022-11-23T01:13:07.3641583Z 96805775a692: Pulling fs layer
2022-11-23T01:13:07.3641827Z 8a7f50c8b503: Waiting
2022-11-23T01:13:07.3642059Z 75f1ead35ace: Pulling fs layer
2022-11-23T01:13:07.3642311Z 6ff0fc00b0a9: Waiting
2022-11-23T01:13:07.3642544Z 96e251412f4d: Waiting
2022-11-23T01:13:07.3642752Z 96805775a692: Waiting
2022-11-23T01:13:07.3642980Z 49d40c00cf56: Waiting
2022-11-23T01:13:07.3643229Z 793c37004dab: Pulling fs layer
2022-11-23T01:13:07.3643475Z cadc5661750d: Pulling fs layer
2022-11-23T01:13:07.3643724Z 7e2d6313145f: Waiting
2022-11-23T01:13:07.3643952Z 793c37004dab: Waiting
2022-11-23T01:13:07.3644189Z 75f1ead35ace: Waiting
2022-11-23T01:13:07.3644418Z 6386b2adbe28: Pulling fs layer
2022-11-23T01:13:07.3644682Z 74aa250bc82f: Pulling fs layer
2022-11-23T01:13:07.3644933Z 6386b2adbe28: Waiting
2022-11-23T01:13:07.3645167Z 436525efe61d: Pulling fs layer
2022-11-23T01:13:07.3645418Z a47cba6c334e: Waiting
2022-11-23T01:13:07.3645672Z 596be1fe0bda: Pulling fs layer
2022-11-23T01:13:07.3645907Z 436525efe61d: Waiting
2022-11-23T01:13:07.3646160Z 772fa4efddc3: Pulling fs layer
2022-11-23T01:13:07.3646426Z 91ddf385377b: Pulling fs layer
2022-11-23T01:13:07.3646676Z 9f7cfb895784: Pulling fs layer
2022-11-23T01:13:07.3646927Z ae48b7377a0d: Waiting
2022-11-23T01:13:07.3647165Z 91ddf385377b: Waiting
2022-11-23T01:13:07.3647396Z 8b8218af0479: Pulling fs layer
2022-11-23T01:13:07.3647643Z 8b8218af0479: Waiting
2022-11-23T01:13:07.3647877Z 74aa250bc82f: Waiting
2022-11-23T01:13:07.3648094Z a58b9ed071f4: Waiting
2022-11-23T01:13:07.3648332Z 772fa4efddc3: Waiting
2022-11-23T01:13:07.3648577Z 9f7cfb895784: Waiting
2022-11-23T01:13:07.4935506Z c58c079e9b17: Download complete
2022-11-23T01:13:07.6080942Z 888240790290: Verifying Checksum
2022-11-23T01:13:07.6081253Z 888240790290: Download complete
2022-11-23T01:13:07.6540357Z e5b80b8bbe91: Verifying Checksum
2022-11-23T01:13:07.6540641Z e5b80b8bbe91: Download complete
2022-11-23T01:13:07.6823994Z a404e5416296: Download complete
2022-11-23T01:13:07.6839670Z 515fe5e34eb4: Download complete
2022-11-23T01:13:07.7577817Z f6e1a56cb32d: Download complete
2022-11-23T01:13:07.8312375Z 304d3c6c28d0: Verifying Checksum
2022-11-23T01:13:07.8312745Z 304d3c6c28d0: Download complete
2022-11-23T01:13:07.9118513Z fac00e927cfe: Download complete
2022-11-23T01:13:08.4510407Z a404e5416296: Pull complete
2022-11-23T01:13:08.7336981Z c58c079e9b17: Pull complete
2022-11-23T01:13:09.2454210Z e5b80b8bbe91: Pull complete
2022-11-23T01:13:09.3957558Z 888240790290: Pull complete
2022-11-23T01:13:09.5131961Z 515fe5e34eb4: Pull complete
2022-11-23T01:13:09.9634516Z f0158c8d8420: Verifying Checksum
2022-11-23T01:13:09.9634998Z f0158c8d8420: Download complete
2022-11-23T01:13:10.0580799Z 3ceac802dd07: Download complete
2022-11-23T01:13:10.1434066Z 0d0e625ba887: Verifying Checksum
2022-11-23T01:13:10.1434573Z 0d0e625ba887: Download complete
2022-11-23T01:13:10.2562522Z bc2be817cb7e: Verifying Checksum
2022-11-23T01:13:10.2562860Z bc2be817cb7e: Download complete
2022-11-23T01:13:10.9918719Z 11eb2106b948: Verifying Checksum
2022-11-23T01:13:10.9919092Z 11eb2106b948: Download complete
2022-11-23T01:13:11.0695284Z 34fa4193c7a6: Verifying Checksum
2022-11-23T01:13:11.0695927Z 34fa4193c7a6: Download complete
2022-11-23T01:13:11.1493703Z a7cf5b3894f8: Verifying Checksum
2022-11-23T01:13:11.1494029Z a7cf5b3894f8: Download complete
2022-11-23T01:13:18.8678493Z 4e4521f12f5a: Verifying Checksum
2022-11-23T01:13:18.8678852Z 4e4521f12f5a: Download complete
2022-11-23T01:13:18.9441104Z 73a2b1f75a3d: Verifying Checksum
2022-11-23T01:13:18.9441429Z 73a2b1f75a3d: Download complete
2022-11-23T01:13:19.0033042Z ba6235196410: Verifying Checksum
2022-11-23T01:13:19.0033438Z ba6235196410: Download complete
2022-11-23T01:13:19.1008407Z 879cdaf83543: Download complete
2022-11-23T01:13:19.1661104Z 6ff0fc00b0a9: Verifying Checksum
2022-11-23T01:13:19.1661446Z 6ff0fc00b0a9: Download complete
2022-11-23T01:13:19.2437616Z a58b9ed071f4: Download complete
2022-11-23T01:13:19.3166895Z a8c562f6a1cf: Download complete
2022-11-23T01:13:20.2619181Z 0a39b4492650: Verifying Checksum
2022-11-23T01:13:20.2619545Z 0a39b4492650: Download complete
2022-11-23T01:13:20.3372403Z 9088ff8de269: Verifying Checksum
2022-11-23T01:13:20.3372804Z 9088ff8de269: Download complete
2022-11-23T01:13:20.4076257Z 165006759af3: Download complete
2022-11-23T01:13:20.4841938Z ae48b7377a0d: Verifying Checksum
2022-11-23T01:13:20.4842568Z ae48b7377a0d: Download complete
2022-11-23T01:13:20.5514947Z b18965f4b6f1: Verifying Checksum
2022-11-23T01:13:20.5515543Z b18965f4b6f1: Download complete
2022-11-23T01:13:20.6221496Z 102ddcd90753: Verifying Checksum
2022-11-23T01:13:20.6222133Z 102ddcd90753: Download complete
2022-11-23T01:13:22.0515318Z c29b96e36bd0: Verifying Checksum
2022-11-23T01:13:22.1271854Z c29b96e36bd0: Download complete
2022-11-23T01:13:22.1272194Z 8a7f50c8b503: Verifying Checksum
2022-11-23T01:13:22.1272487Z 8a7f50c8b503: Download complete
2022-11-23T01:13:22.2048918Z 863c35620b44: Download complete
2022-11-23T01:13:22.5954306Z 183e4209dc37: Verifying Checksum
2022-11-23T01:13:22.5954927Z 183e4209dc37: Download complete
2022-11-23T01:13:22.6034322Z 5f5dd1cba120: Verifying Checksum
2022-11-23T01:13:22.6034618Z 5f5dd1cba120: Download complete
2022-11-23T01:13:22.6513545Z a47cba6c334e: Download complete
2022-11-23T01:13:22.6927792Z a9f3d4742233: Verifying Checksum
2022-11-23T01:13:22.6928119Z a9f3d4742233: Download complete
2022-11-23T01:13:22.7702778Z 023a41fa48e6: Verifying Checksum
2022-11-23T01:13:22.7703107Z 023a41fa48e6: Download complete
2022-11-23T01:13:22.9095032Z 3cefa8a4607f: Verifying Checksum
2022-11-23T01:13:22.9095396Z 3cefa8a4607f: Download complete
2022-11-23T01:13:22.9951434Z 49d40c00cf56: Download complete
2022-11-23T01:13:23.0723127Z 7e2d6313145f: Verifying Checksum
2022-11-23T01:13:23.0723484Z 7e2d6313145f: Download complete
2022-11-23T01:13:23.2238077Z 96e251412f4d: Verifying Checksum
2022-11-23T01:13:23.2238436Z 96e251412f4d: Download complete
2022-11-23T01:13:23.3034432Z 75f1ead35ace: Verifying Checksum
2022-11-23T01:13:23.3034776Z 75f1ead35ace: Download complete
2022-11-23T01:13:23.3634141Z 793c37004dab: Verifying Checksum
2022-11-23T01:13:23.3634456Z 793c37004dab: Download complete
2022-11-23T01:13:23.4264139Z cadc5661750d: Verifying Checksum
2022-11-23T01:13:23.4264474Z cadc5661750d: Download complete
2022-11-23T01:13:23.4846781Z 6386b2adbe28: Verifying Checksum
2022-11-23T01:13:23.4847222Z 6386b2adbe28: Download complete
2022-11-23T01:13:23.6613323Z 74aa250bc82f: Verifying Checksum
2022-11-23T01:13:23.6613668Z 74aa250bc82f: Download complete
2022-11-23T01:13:23.7367518Z 436525efe61d: Verifying Checksum
2022-11-23T01:13:23.7368160Z 436525efe61d: Download complete
2022-11-23T01:13:24.3325113Z 596be1fe0bda: Verifying Checksum
2022-11-23T01:13:24.3325496Z 596be1fe0bda: Download complete
2022-11-23T01:13:24.4139514Z 772fa4efddc3: Verifying Checksum
2022-11-23T01:13:24.4140484Z 772fa4efddc3: Download complete
2022-11-23T01:13:27.7253207Z 96805775a692: Verifying Checksum
2022-11-23T01:13:27.7253544Z 96805775a692: Download complete
2022-11-23T01:13:27.7894667Z 9f7cfb895784: Verifying Checksum
2022-11-23T01:13:27.7895272Z 9f7cfb895784: Download complete
2022-11-23T01:13:27.8757835Z 8b8218af0479: Verifying Checksum
2022-11-23T01:13:27.8758408Z 8b8218af0479: Download complete
2022-11-23T01:13:30.9512882Z 3f6b06edd3f5: Verifying Checksum
2022-11-23T01:13:30.9513219Z 3f6b06edd3f5: Download complete
2022-11-23T01:13:32.7083501Z 4e4521f12f5a: Pull complete
2022-11-23T01:13:32.8377215Z f6e1a56cb32d: Pull complete
2022-11-23T01:13:54.1638031Z c29b96e36bd0: Pull complete
2022-11-23T01:13:54.3948053Z 91ddf385377b: Verifying Checksum
2022-11-23T01:13:54.3948377Z 91ddf385377b: Download complete
2022-11-23T01:13:56.0157197Z 304d3c6c28d0: Pull complete
2022-11-23T01:13:57.8868291Z fac00e927cfe: Pull complete
2022-11-23T01:14:05.4792248Z f0158c8d8420: Pull complete
2022-11-23T01:14:07.3265511Z 3ceac802dd07: Pull complete
2022-11-23T01:14:09.2038672Z 0d0e625ba887: Pull complete
2022-11-23T01:14:11.3610683Z bc2be817cb7e: Pull complete
2022-11-23T01:14:15.6107728Z 11eb2106b948: Pull complete
2022-11-23T01:14:18.2592441Z 34fa4193c7a6: Pull complete
2022-11-23T01:14:20.1389686Z a7cf5b3894f8: Pull complete
2022-11-23T01:14:56.0872203Z 3f6b06edd3f5: Pull complete
2022-11-23T01:14:57.8195819Z 73a2b1f75a3d: Pull complete
2022-11-23T01:14:57.9395024Z ba6235196410: Pull complete
2022-11-23T01:14:58.0475142Z 879cdaf83543: Pull complete
2022-11-23T01:14:58.1391178Z 6ff0fc00b0a9: Pull complete
2022-11-23T01:14:58.2485556Z a58b9ed071f4: Pull complete
2022-11-23T01:14:58.3724261Z a8c562f6a1cf: Pull complete
2022-11-23T01:15:00.7541271Z 0a39b4492650: Pull complete
2022-11-23T01:15:00.8626731Z 9088ff8de269: Pull complete
2022-11-23T01:15:00.9675239Z 165006759af3: Pull complete
2022-11-23T01:15:01.1049192Z ae48b7377a0d: Pull complete
2022-11-23T01:15:01.2407145Z b18965f4b6f1: Pull complete
2022-11-23T01:15:01.3427254Z 102ddcd90753: Pull complete
2022-11-23T01:15:08.6509885Z 5f5dd1cba120: Pull complete
2022-11-23T01:15:10.5570424Z 8a7f50c8b503: Pull complete
2022-11-23T01:15:12.4321246Z 863c35620b44: Pull complete
2022-11-23T01:15:15.3037692Z 183e4209dc37: Pull complete
2022-11-23T01:15:17.3974366Z a47cba6c334e: Pull complete
2022-11-23T01:15:20.2953593Z a9f3d4742233: Pull complete
2022-11-23T01:15:24.1956105Z 3cefa8a4607f: Pull complete
2022-11-23T01:15:26.9413930Z 023a41fa48e6: Pull complete
2022-11-23T01:15:31.5345700Z 96e251412f4d: Pull complete
2022-11-23T01:15:33.8699389Z 49d40c00cf56: Pull complete
2022-11-23T01:15:35.9408543Z 7e2d6313145f: Pull complete
2022-11-23T01:15:44.9416868Z 96805775a692: Pull complete
2022-11-23T01:15:46.5824150Z 75f1ead35ace: Pull complete
2022-11-23T01:15:48.4963281Z 793c37004dab: Pull complete
2022-11-23T01:15:50.4032515Z cadc5661750d: Pull complete
2022-11-23T01:15:52.2492445Z 6386b2adbe28: Pull complete
2022-11-23T01:15:55.5153383Z 74aa250bc82f: Pull complete
2022-11-23T01:15:57.4554772Z 436525efe61d: Pull complete
2022-11-23T01:15:59.6312346Z 596be1fe0bda: Pull complete
2022-11-23T01:15:59.7379816Z 772fa4efddc3: Pull complete
2022-11-23T01:16:41.6259273Z 91ddf385377b: Pull complete
2022-11-23T01:16:43.5359702Z 9f7cfb895784: Pull complete
2022-11-23T01:16:45.4135820Z 8b8218af0479: Pull complete
2022-11-23T01:16:46.6981644Z Digest: sha256:3a5626edfb2c43fb24303351be75287af92426b6bb7c6df2defc98f980346c6a
2022-11-23T01:16:47.1987407Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:16:47.4820534Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:16:47.4922081Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2022-11-23T01:16:47.4922440Z with:
2022-11-23T01:16:47.4922685Z   driver-version: 515.76
2022-11-23T01:16:47.4922907Z env:
2022-11-23T01:16:47.4923147Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:16:47.4923408Z ##[endgroup]
2022-11-23T01:16:47.6122145Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-11-23T01:16:47.6122445Z with:
2022-11-23T01:16:47.6122680Z   timeout_minutes: 10
2022-11-23T01:16:47.6122935Z   max_attempts: 3
2022-11-23T01:16:47.6128891Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x

        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms

        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            sudo yum groupinstall -y "Development Tools"
            # ensure our kernel install is the same as our underlying kernel,
            # groupinstall "Development Tools" has a habit of mismatching kernel headers
            sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
            sudo modprobe backlight
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi

        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e
            nvidia-smi
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-11-23T01:16:47.6135058Z   retry_wait_seconds: 10
2022-11-23T01:16:47.6135320Z   polling_interval_seconds: 1
2022-11-23T01:16:47.6135594Z   warning_on_retry: true
2022-11-23T01:16:47.6135858Z   continue_on_error: false
2022-11-23T01:16:47.6136081Z env:
2022-11-23T01:16:47.6136324Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:16:47.6136589Z   DRIVER_VERSION: 515.76
2022-11-23T01:16:47.6136821Z ##[endgroup]
2022-11-23T01:16:47.6695566Z 
2022-11-23T01:16:47.6718185Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:16:47.6759942Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run ==
2022-11-23T01:16:47.6761836Z + sudo yum remove -y nvidia-driver-latest-dkms
2022-11-23T01:16:48.1453177Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:48.1888352Z No Match for argument: nvidia-driver-latest-dkms
2022-11-23T01:16:48.2180110Z No Packages marked for removal
2022-11-23T01:16:48.2333304Z + echo 'Before installing NVIDIA driver'
2022-11-23T01:16:48.2333582Z + lspci
2022-11-23T01:16:48.2333840Z Before installing NVIDIA driver
2022-11-23T01:16:49.4748237Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:16:49.4749119Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:16:49.4749549Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:16:49.4749949Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:16:49.4751886Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:16:49.4752400Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:16:49.4753082Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:49.4753608Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:49.4754015Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:16:49.4754335Z + lsmod
2022-11-23T01:16:49.4770707Z Module                  Size  Used by
2022-11-23T01:16:49.4771079Z xt_conntrack           16384  1
2022-11-23T01:16:49.4771358Z ipt_MASQUERADE         16384  1
2022-11-23T01:16:49.4771671Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:16:49.4771955Z nf_conntrack_netlink    49152  0
2022-11-23T01:16:49.4772267Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:16:49.4772551Z xfrm_user              45056  1
2022-11-23T01:16:49.4772798Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:16:49.4773064Z xt_addrtype            16384  2
2022-11-23T01:16:49.4773331Z iptable_filter         16384  1
2022-11-23T01:16:49.4773583Z iptable_nat            16384  1
2022-11-23T01:16:49.4773855Z nf_conntrack_ipv4      16384  3
2022-11-23T01:16:49.4774157Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:16:49.4774443Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:16:49.4774771Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:16:49.4777649Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:16:49.4778467Z br_netfilter           24576  0
2022-11-23T01:16:49.4778768Z bridge                172032  1 br_netfilter
2022-11-23T01:16:49.4779046Z stp                    16384  1 bridge
2022-11-23T01:16:49.4779297Z llc                    16384  2 bridge,stp
2022-11-23T01:16:49.4779559Z overlay                86016  0
2022-11-23T01:16:49.4779816Z sunrpc                393216  1
2022-11-23T01:16:49.4780056Z dm_mirror              28672  0
2022-11-23T01:16:49.4780325Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:16:49.4780643Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:16:49.4780940Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:16:49.4781212Z dax                    69632  1 dm_mod
2022-11-23T01:16:49.4781469Z sb_edac                24576  0
2022-11-23T01:16:49.4781726Z crc32_pclmul           16384  0
2022-11-23T01:16:49.4781973Z ghash_clmulni_intel    16384  0
2022-11-23T01:16:49.4782243Z pcbc                   16384  0
2022-11-23T01:16:49.4782496Z aesni_intel           188416  0
2022-11-23T01:16:49.4787407Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:16:49.4788072Z ata_piix               36864  0
2022-11-23T01:16:49.4788643Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:16:49.4789281Z glue_helper            16384  1 aesni_intel
2022-11-23T01:16:49.4789858Z pcc_cpufreq            16384  0
2022-11-23T01:16:49.4790424Z libata                266240  1 ata_piix
2022-11-23T01:16:49.4790781Z mousedev               24576  0
2022-11-23T01:16:49.4791107Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:16:49.4791433Z evdev                  20480  3
2022-11-23T01:16:49.4791693Z scsi_mod              245760  1 libata
2022-11-23T01:16:49.4791954Z psmouse                32768  0
2022-11-23T01:16:49.4792187Z button                 16384  0
2022-11-23T01:16:49.4792431Z ena                   114688  0
2022-11-23T01:16:49.4792681Z xen_blkfront           49152  2
2022-11-23T01:16:49.4792923Z crc32c_intel           24576  0
2022-11-23T01:16:49.4793174Z autofs4                49152  2
2022-11-23T01:16:49.4793421Z + modinfo nvidia
2022-11-23T01:16:49.4793680Z modinfo: ERROR: Module nvidia not found.
2022-11-23T01:16:49.4793944Z + true
2022-11-23T01:16:49.4794178Z + HAS_NVIDIA_DRIVER=0
2022-11-23T01:16:49.4794520Z ++ command -v nvidia-smi
2022-11-23T01:16:49.4794808Z + '[' -x '' ']'
2022-11-23T01:16:49.4795077Z + '[' 0 -eq 0 ']'
2022-11-23T01:16:49.4795405Z + sudo yum groupinstall -y 'Development Tools'
2022-11-23T01:16:49.9520670Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:50.2502474Z Resolving Dependencies
2022-11-23T01:16:50.2507031Z --> Running transaction check
2022-11-23T01:16:50.2510063Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2022-11-23T01:16:50.2724853Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2022-11-23T01:16:50.4867594Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2022-11-23T01:16:50.4869214Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2022-11-23T01:16:50.4915384Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2022-11-23T01:16:50.4922159Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2022-11-23T01:16:50.4933162Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2022-11-23T01:16:50.5050158Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2022-11-23T01:16:50.5057167Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2022-11-23T01:16:50.5101992Z --> Processing Dependency: emacs-filesystem for package: cscope-15.8-10.amzn2.0.2.x86_64
2022-11-23T01:16:50.5125872Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2022-11-23T01:16:50.5134613Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2022-11-23T01:16:50.5142586Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2022-11-23T01:16:50.5244600Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2022-11-23T01:16:50.5378658Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2022-11-23T01:16:50.5398000Z ---> Package gcc.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.5571531Z --> Processing Dependency: cpp = 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.5592061Z --> Processing Dependency: libsanitizer >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.5647475Z --> Processing Dependency: libquadmath >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.5700527Z --> Processing Dependency: libmpx >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.5756020Z --> Processing Dependency: libitm >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.5809011Z --> Processing Dependency: libcilkrts >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.5863090Z --> Processing Dependency: libatomic >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.5918178Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.6081097Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.6102911Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.6124073Z ---> Package gcc-c++.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.6151113Z ---> Package gcc-gfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.6185435Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:50.6248102Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2022-11-23T01:16:50.6262883Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2022-11-23T01:16:50.6314377Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2022-11-23T01:16:50.6329486Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2022-11-23T01:16:50.6348488Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2022-11-23T01:16:50.6377703Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2022-11-23T01:16:50.6414557Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2022-11-23T01:16:50.6439826Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2022-11-23T01:16:50.6472070Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-11-23T01:16:50.6712180Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2022-11-23T01:16:50.6729932Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-11-23T01:16:50.6753555Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-11-23T01:16:50.6926409Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6946112Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6947355Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6948708Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6950071Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6951440Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6952785Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6954123Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6955504Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6956256Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6956853Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6957464Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6958082Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6958675Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6959296Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6959898Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6975080Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.6996137Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:50.7020608Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2022-11-23T01:16:50.7041475Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 will be installed
2022-11-23T01:16:50.7089019Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-11-23T01:16:50.7106687Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-11-23T01:16:50.7119223Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-11-23T01:16:50.7296230Z ---> Package systemtap.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-11-23T01:16:50.7309627Z --> Processing Dependency: systemtap-devel = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:50.7325063Z --> Processing Dependency: systemtap-client = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:50.7338672Z --> Running transaction check
2022-11-23T01:16:50.7342405Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2022-11-23T01:16:50.7423767Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-11-23T01:16:50.7462370Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2022-11-23T01:16:50.7476178Z ---> Package cpp.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.7549149Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2022-11-23T01:16:50.7559776Z ---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
2022-11-23T01:16:50.7560786Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2022-11-23T01:16:50.7632683Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2022-11-23T01:16:50.7691278Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2022-11-23T01:16:50.7700237Z ---> Package glibc-devel.x86_64 0:2.26-62.amzn2 will be installed
2022-11-23T01:16:50.7822218Z --> Processing Dependency: glibc-headers = 2.26-62.amzn2 for package: glibc-devel-2.26-62.amzn2.x86_64
2022-11-23T01:16:50.7850399Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-62.amzn2.x86_64
2022-11-23T01:16:50.7851605Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2 will be installed
2022-11-23T01:16:50.7856161Z ---> Package libatomic.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.7869807Z ---> Package libcilkrts.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.7896242Z ---> Package libgfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.7931686Z ---> Package libitm.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.7948161Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2022-11-23T01:16:50.7960309Z ---> Package libmpx.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.7975241Z ---> Package libquadmath.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.8000511Z ---> Package libsanitizer.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:50.8045802Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2022-11-23T01:16:50.8061062Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2022-11-23T01:16:50.8081884Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2022-11-23T01:16:50.8157430Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:50.8196037Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:50.8197898Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:50.8216416Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:50.8234118Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:50.8240329Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2022-11-23T01:16:50.8288042Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2022-11-23T01:16:50.8383005Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2022-11-23T01:16:50.8394848Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2022-11-23T01:16:50.8410507Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2022-11-23T01:16:50.8411589Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-11-23T01:16:50.8439943Z ---> Package systemtap-client.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-11-23T01:16:50.8648959Z --> Processing Dependency: mokutil for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:50.8663525Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:50.8689850Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:50.8690975Z ---> Package systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-11-23T01:16:50.8807146Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:50.9830484Z --> Running transaction check
2022-11-23T01:16:50.9831495Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-11-23T01:16:50.9841140Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2022-11-23T01:16:50.9867588Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2022-11-23T01:16:50.9868681Z ---> Package glibc-headers.x86_64 0:2.26-62.amzn2 will be installed
2022-11-23T01:16:50.9944467Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-62.amzn2.x86_64
2022-11-23T01:16:51.1045905Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-62.amzn2.x86_64
2022-11-23T01:16:51.1046831Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2022-11-23T01:16:51.1113053Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2022-11-23T01:16:51.1139348Z ---> Package kernel-devel.x86_64 0:4.14.296-222.539.amzn2 will be installed
2022-11-23T01:16:51.1166326Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.296-222.539.amzn2.x86_64
2022-11-23T01:16:51.1186154Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2022-11-23T01:16:51.1214344Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2022-11-23T01:16:51.1232574Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2022-11-23T01:16:51.1281676Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-11-23T01:16:51.1302424Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-11-23T01:16:51.1303705Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-11-23T01:16:51.1304667Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2022-11-23T01:16:51.1317377Z --> Running transaction check
2022-11-23T01:16:51.1318266Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2022-11-23T01:16:51.1335434Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2022-11-23T01:16:51.1348115Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2022-11-23T01:16:51.1375947Z ---> Package kernel-headers.x86_64 0:4.14.296-222.539.amzn2 will be installed
2022-11-23T01:16:51.1377360Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2022-11-23T01:16:51.1394882Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2022-11-23T01:16:51.1451796Z --> Running transaction check
2022-11-23T01:16:51.1452728Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2 will be installed
2022-11-23T01:16:51.4055818Z --> Finished Dependency Resolution
2022-11-23T01:16:51.4809622Z 
2022-11-23T01:16:51.4810559Z Dependencies Resolved
2022-11-23T01:16:51.4924806Z 
2022-11-23T01:16:51.4925228Z ================================================================================
2022-11-23T01:16:51.4925933Z  Package                 Arch     Version                    Repository    Size
2022-11-23T01:16:51.4926598Z ================================================================================
2022-11-23T01:16:51.4927339Z Installing for group install "Development Tools":
2022-11-23T01:16:51.4928436Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2022-11-23T01:16:51.4929255Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2022-11-23T01:16:51.4930247Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2022-11-23T01:16:51.4930786Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2022-11-23T01:16:51.4931223Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2022-11-23T01:16:51.4931920Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2022-11-23T01:16:51.4932359Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2022-11-23T01:16:51.4932806Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2022-11-23T01:16:51.4933219Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2022-11-23T01:16:51.4933652Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2022-11-23T01:16:51.4934075Z  gcc                     x86_64   7.3.1-15.amzn2             amzn2-core    22 M
2022-11-23T01:16:51.4934497Z  gcc-c++                 x86_64   7.3.1-15.amzn2             amzn2-core    13 M
2022-11-23T01:16:51.4934916Z  gcc-gfortran            x86_64   7.3.1-15.amzn2             amzn2-core    11 M
2022-11-23T01:16:51.4935352Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2022-11-23T01:16:51.4935786Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2022-11-23T01:16:51.4936205Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2022-11-23T01:16:51.4936635Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2022-11-23T01:16:51.4937064Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2022-11-23T01:16:51.4937495Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2022-11-23T01:16:51.4938356Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2022-11-23T01:16:51.4938842Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2022-11-23T01:16:51.4939287Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2022-11-23T01:16:51.4939695Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2022-11-23T01:16:51.4940206Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.14        amzn2-core    90 k
2022-11-23T01:16:51.4940666Z  systemtap               x86_64   4.5-1.amzn2.0.1            amzn2-core    12 k
2022-11-23T01:16:51.4940987Z Installing for dependencies:
2022-11-23T01:16:51.4941386Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2022-11-23T01:16:51.4941824Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2022-11-23T01:16:51.4942272Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2022-11-23T01:16:51.4942721Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2022-11-23T01:16:51.4943138Z  cpp                     x86_64   7.3.1-15.amzn2             amzn2-core   9.2 M
2022-11-23T01:16:51.4943561Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2022-11-23T01:16:51.4943992Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2022-11-23T01:16:51.4944437Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2022-11-23T01:16:51.4944922Z  emacs-filesystem        noarch   1:27.2-4.amzn2.0.1         amzn2-core    67 k
2022-11-23T01:16:51.4945368Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2022-11-23T01:16:51.4945867Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2022-11-23T01:16:51.4946557Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2022-11-23T01:16:51.4947002Z  glibc-devel             x86_64   2.26-62.amzn2              amzn2-core   995 k
2022-11-23T01:16:51.4947452Z  glibc-headers           x86_64   2.26-62.amzn2              amzn2-core   516 k
2022-11-23T01:16:51.4947873Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2022-11-23T01:16:51.4948316Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.2        amzn2-core    23 k
2022-11-23T01:16:51.4948780Z  kernel-devel            x86_64   4.14.296-222.539.amzn2     amzn2-core    13 M
2022-11-23T01:16:51.4949431Z  kernel-headers          x86_64   4.14.296-222.539.amzn2     amzn2-core   1.2 M
2022-11-23T01:16:51.4949870Z  libatomic               x86_64   7.3.1-15.amzn2             amzn2-core    46 k
2022-11-23T01:16:51.4950302Z  libcilkrts              x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-11-23T01:16:51.4950739Z  libgfortran             x86_64   7.3.1-15.amzn2             amzn2-core   536 k
2022-11-23T01:16:51.4951150Z  libitm                  x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-11-23T01:16:51.4951583Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2022-11-23T01:16:51.4952016Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2022-11-23T01:16:51.4952445Z  libmpx                  x86_64   7.3.1-15.amzn2             amzn2-core    51 k
2022-11-23T01:16:51.4952860Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2022-11-23T01:16:51.4953297Z  libquadmath             x86_64   7.3.1-15.amzn2             amzn2-core   189 k
2022-11-23T01:16:51.4953737Z  libsanitizer            x86_64   7.3.1-15.amzn2             amzn2-core   642 k
2022-11-23T01:16:51.4954141Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2022-11-23T01:16:51.4954557Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2022-11-23T01:16:51.4954980Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2022-11-23T01:16:51.4955399Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2022-11-23T01:16:51.4955810Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2022-11-23T01:16:51.4956263Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2022-11-23T01:16:51.4956732Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2022-11-23T01:16:51.4957189Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2022-11-23T01:16:51.4957676Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2022-11-23T01:16:51.4958144Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2022-11-23T01:16:51.4958615Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2022-11-23T01:16:51.4959052Z  systemtap-client        x86_64   4.5-1.amzn2.0.1            amzn2-core   3.7 M
2022-11-23T01:16:51.4959503Z  systemtap-devel         x86_64   4.5-1.amzn2.0.1            amzn2-core   2.3 M
2022-11-23T01:16:51.4959945Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2022-11-23T01:16:51.4960357Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.2         amzn2-core    50 k
2022-11-23T01:16:51.4960570Z 
2022-11-23T01:16:51.4960688Z Transaction Summary
2022-11-23T01:16:51.4960977Z ================================================================================
2022-11-23T01:16:51.4961303Z Install  25 Packages (+43 Dependent packages)
2022-11-23T01:16:51.4961490Z 
2022-11-23T01:16:51.4961623Z Total download size: 96 M
2022-11-23T01:16:51.4961892Z Installed size: 303 M
2022-11-23T01:16:51.4962160Z Downloading packages:
2022-11-23T01:16:51.4976357Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-11-23T01:16:52.9296335Z --------------------------------------------------------------------------------
2022-11-23T01:16:52.9296794Z Total                                               67 MB/s |  96 MB  00:01     
2022-11-23T01:16:53.0344793Z Running transaction check
2022-11-23T01:16:53.1104950Z Running transaction test
2022-11-23T01:16:55.4129174Z Transaction test succeeded
2022-11-23T01:16:55.4132079Z Running transaction
2022-11-23T01:17:00.6839570Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/68 
2022-11-23T01:17:03.1634063Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/68 
2022-11-23T01:17:06.0523158Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/68 
2022-11-23T01:17:09.0527302Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/68 
2022-11-23T01:17:10.5360273Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/68 
2022-11-23T01:17:12.9744958Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/68 
2022-11-23T01:17:15.4877599Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/68 
2022-11-23T01:17:16.5697444Z   Installing : libquadmath-7.3.1-15.amzn2.x86_64                           8/68 
2022-11-23T01:17:16.5945573Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/68 
2022-11-23T01:17:16.6773143Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/68 
2022-11-23T01:17:17.7388605Z   Installing : libgfortran-7.3.1-15.amzn2.x86_64                          11/68 
2022-11-23T01:17:17.7945513Z   Installing : cpp-7.3.1-15.amzn2.x86_64                                  12/68 
2022-11-23T01:17:17.8400030Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         13/68 
2022-11-23T01:17:17.9100361Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        14/68 
2022-11-23T01:17:17.9702342Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   15/68 
2022-11-23T01:17:18.0818956Z   Installing : elfutils-0.176-2.amzn2.x86_64                              16/68 
2022-11-23T01:17:18.1135947Z   Installing : libsanitizer-7.3.1-15.amzn2.x86_64                         17/68 
2022-11-23T01:17:18.1555379Z   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 18/68 
2022-11-23T01:17:18.1895335Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          19/68 
2022-11-23T01:17:18.2197532Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        20/68 
2022-11-23T01:17:18.3028869Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               21/68 
2022-11-23T01:17:18.3572515Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      22/68 
2022-11-23T01:17:18.4503754Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                23/68 
2022-11-23T01:17:18.6340409Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         24/68 
2022-11-23T01:17:18.6751047Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           25/68 
2022-11-23T01:17:19.0718126Z   Installing : libitm-7.3.1-15.amzn2.x86_64                               26/68 
2022-11-23T01:17:19.1013074Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              27/68 
2022-11-23T01:17:19.1311196Z   Installing : libmpx-7.3.1-15.amzn2.x86_64                               28/68 
2022-11-23T01:17:19.1492425Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      29/68 
2022-11-23T01:17:19.1813375Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  30/68 
2022-11-23T01:17:19.2122577Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               31/68 
2022-11-23T01:17:19.3085525Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  32/68 
2022-11-23T01:17:19.3996402Z   Installing : autoconf-2.69-11.amzn2.noarch                              33/68 
2022-11-23T01:17:19.5082491Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      34/68 
2022-11-23T01:17:19.5489754Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           35/68 
2022-11-23T01:17:19.5731897Z   Installing : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       36/68 
2022-11-23T01:17:19.5956319Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 37/68 
2022-11-23T01:17:19.8919863Z   Installing : libatomic-7.3.1-15.amzn2.x86_64                            38/68 
2022-11-23T01:17:20.0646133Z   Installing : kernel-headers-4.14.296-222.539.amzn2.x86_64               39/68 
2022-11-23T01:17:20.2021940Z   Installing : glibc-headers-2.26-62.amzn2.x86_64                         40/68 
2022-11-23T01:17:20.2457445Z   Installing : glibc-devel-2.26-62.amzn2.x86_64                           41/68 
2022-11-23T01:17:22.3000832Z   Installing : libcilkrts-7.3.1-15.amzn2.x86_64                           42/68 
2022-11-23T01:17:26.1641033Z   Installing : gcc-7.3.1-15.amzn2.x86_64                                  43/68 
2022-11-23T01:17:37.2116736Z   Installing : kernel-devel-4.14.296-222.539.amzn2.x86_64                 44/68 
2022-11-23T01:17:37.8330005Z   Installing : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     45/68 
2022-11-23T01:17:37.8960015Z   Installing : systemtap-client-4.5-1.amzn2.0.1.x86_64                    46/68 
2022-11-23T01:17:37.9544910Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           47/68 
2022-11-23T01:17:38.0932565Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             48/68 
2022-11-23T01:17:38.2790150Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 49/68 
2022-11-23T01:17:38.3734686Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-11-23T01:17:39.5857189Z   Installing : systemtap-4.5-1.amzn2.0.1.x86_64                           51/68 
2022-11-23T01:17:41.2036916Z   Installing : gcc-gfortran-7.3.1-15.amzn2.x86_64                         52/68 
2022-11-23T01:17:41.3308688Z   Installing : gcc-c++-7.3.1-15.amzn2.x86_64                              53/68 
2022-11-23T01:17:41.3736911Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        54/68 
2022-11-23T01:17:41.4122192Z   Installing : intltool-0.50.2-7.amzn2.noarch                             55/68 
2022-11-23T01:17:41.4692822Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       56/68 
2022-11-23T01:17:41.5301751Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            57/68 
2022-11-23T01:17:41.6394968Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             58/68 
2022-11-23T01:17:41.7044128Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             59/68 
2022-11-23T01:17:41.7517738Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               60/68 
2022-11-23T01:17:41.7958024Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              61/68 
2022-11-23T01:17:41.8483906Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          62/68 
2022-11-23T01:17:42.5442354Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        63/68 
2022-11-23T01:17:42.6013688Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             64/68 
2022-11-23T01:17:42.6625657Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           65/68 
2022-11-23T01:17:42.9850815Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      66/68 
2022-11-23T01:17:43.0286324Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            67/68 
2022-11-23T01:17:43.0941970Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        68/68 
2022-11-23T01:17:43.1156080Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                  1/68 
2022-11-23T01:17:43.1256995Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/68 
2022-11-23T01:17:43.1386289Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/68 
2022-11-23T01:17:43.1485707Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             4/68 
2022-11-23T01:17:43.1646458Z   Verifying  : kernel-devel-4.14.296-222.539.amzn2.x86_64                  5/68 
2022-11-23T01:17:43.1745821Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              6/68 
2022-11-23T01:17:43.1893154Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                            7/68 
2022-11-23T01:17:43.2009916Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         8/68 
2022-11-23T01:17:43.2105324Z   Verifying  : glibc-devel-2.26-62.amzn2.x86_64                            9/68 
2022-11-23T01:17:43.2208653Z   Verifying  : gcc-gfortran-7.3.1-15.amzn2.x86_64                         10/68 
2022-11-23T01:17:43.2327906Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            11/68 
2022-11-23T01:17:43.2451050Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      12/68 
2022-11-23T01:17:43.2548969Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                            13/68 
2022-11-23T01:17:43.2664666Z   Verifying  : libcilkrts-7.3.1-15.amzn2.x86_64                           14/68 
2022-11-23T01:17:43.2772990Z   Verifying  : kernel-headers-4.14.296-222.539.amzn2.x86_64               15/68 
2022-11-23T01:17:43.2875624Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        16/68 
2022-11-23T01:17:43.2970090Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            17/68 
2022-11-23T01:17:43.3079848Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           18/68 
2022-11-23T01:17:43.3186071Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             19/68 
2022-11-23T01:17:43.3289483Z   Verifying  : gcc-c++-7.3.1-15.amzn2.x86_64                              20/68 
2022-11-23T01:17:43.3413698Z   Verifying  : libatomic-7.3.1-15.amzn2.x86_64                            21/68 
2022-11-23T01:17:43.3536355Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               22/68 
2022-11-23T01:17:43.3655069Z   Verifying  : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     23/68 
2022-11-23T01:17:43.3759803Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       24/68 
2022-11-23T01:17:43.3863474Z   Verifying  : glibc-headers-2.26-62.amzn2.x86_64                         25/68 
2022-11-23T01:17:43.3978318Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      26/68 
2022-11-23T01:17:43.4085392Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              27/68 
2022-11-23T01:17:43.4180873Z   Verifying  : libquadmath-7.3.1-15.amzn2.x86_64                          28/68 
2022-11-23T01:17:43.4285904Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             29/68 
2022-11-23T01:17:43.4385708Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          30/68 
2022-11-23T01:17:43.4505202Z   Verifying  : cpp-7.3.1-15.amzn2.x86_64                                  31/68 
2022-11-23T01:17:43.4645867Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       32/68 
2022-11-23T01:17:43.4736907Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  33/68 
2022-11-23T01:17:43.4849607Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  34/68 
2022-11-23T01:17:43.4976398Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      35/68 
2022-11-23T01:17:43.5066729Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           36/68 
2022-11-23T01:17:43.5169925Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 37/68 
2022-11-23T01:17:43.5269076Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           38/68 
2022-11-23T01:17:43.5406945Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      39/68 
2022-11-23T01:17:43.5529142Z   Verifying  : libmpx-7.3.1-15.amzn2.x86_64                               40/68 
2022-11-23T01:17:43.5702368Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          41/68 
2022-11-23T01:17:43.5840607Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             42/68 
2022-11-23T01:17:43.5943270Z   Verifying  : libgfortran-7.3.1-15.amzn2.x86_64                          43/68 
2022-11-23T01:17:43.6057651Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              44/68 
2022-11-23T01:17:43.6158852Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        45/68 
2022-11-23T01:17:43.6280408Z   Verifying  : libitm-7.3.1-15.amzn2.x86_64                               46/68 
2022-11-23T01:17:43.6381149Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        47/68 
2022-11-23T01:17:43.6485683Z   Verifying  : gcc-7.3.1-15.amzn2.x86_64                                  48/68 
2022-11-23T01:17:43.6587236Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          49/68 
2022-11-23T01:17:43.6698585Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-11-23T01:17:43.6819379Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   51/68 
2022-11-23T01:17:43.6948597Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              52/68 
2022-11-23T01:17:43.7062516Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        53/68 
2022-11-23T01:17:43.7197236Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              54/68 
2022-11-23T01:17:43.7331369Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         55/68 
2022-11-23T01:17:43.7461113Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             56/68 
2022-11-23T01:17:43.7570771Z   Verifying  : systemtap-4.5-1.amzn2.0.1.x86_64                           57/68 
2022-11-23T01:17:43.7698279Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                58/68 
2022-11-23T01:17:43.7819663Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               59/68 
2022-11-23T01:17:43.7941070Z   Verifying  : systemtap-client-4.5-1.amzn2.0.1.x86_64                    60/68 
2022-11-23T01:17:43.8063995Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          61/68 
2022-11-23T01:17:43.8188024Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               62/68 
2022-11-23T01:17:43.8310239Z   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 63/68 
2022-11-23T01:17:43.8422983Z   Verifying  : libsanitizer-7.3.1-15.amzn2.x86_64                         64/68 
2022-11-23T01:17:43.8547779Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              65/68 
2022-11-23T01:17:43.8658280Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              66/68 
2022-11-23T01:17:43.8778832Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   67/68 
2022-11-23T01:17:43.9552899Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         68/68 
2022-11-23T01:17:43.9556069Z 
2022-11-23T01:17:43.9556549Z Installed:
2022-11-23T01:17:43.9557049Z   autoconf.noarch 0:2.69-11.amzn2                                               
2022-11-23T01:17:43.9557499Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2022-11-23T01:17:43.9560047Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2022-11-23T01:17:43.9560508Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2022-11-23T01:17:43.9560951Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2022-11-23T01:17:43.9561385Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2022-11-23T01:17:43.9561806Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2022-11-23T01:17:43.9562219Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2022-11-23T01:17:43.9562639Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2022-11-23T01:17:43.9563038Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2022-11-23T01:17:43.9563423Z   gcc.x86_64 0:7.3.1-15.amzn2                                                   
2022-11-23T01:17:43.9563823Z   gcc-c++.x86_64 0:7.3.1-15.amzn2                                               
2022-11-23T01:17:43.9568857Z   gcc-gfortran.x86_64 0:7.3.1-15.amzn2                                          
2022-11-23T01:17:43.9569346Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2022-11-23T01:17:43.9569789Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2022-11-23T01:17:43.9570471Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2022-11-23T01:17:43.9570890Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2022-11-23T01:17:43.9571325Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2022-11-23T01:17:43.9571737Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2022-11-23T01:17:43.9572154Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2022-11-23T01:17:43.9572560Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2022-11-23T01:17:43.9572978Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2022-11-23T01:17:43.9573389Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2022-11-23T01:17:43.9573804Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14                                
2022-11-23T01:17:43.9574265Z   systemtap.x86_64 0:4.5-1.amzn2.0.1                                            
2022-11-23T01:17:43.9574472Z 
2022-11-23T01:17:43.9574594Z Dependency Installed:
2022-11-23T01:17:43.9574978Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2022-11-23T01:17:43.9575367Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2022-11-23T01:17:43.9575788Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2022-11-23T01:17:43.9576216Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2022-11-23T01:17:43.9576662Z   cpp.x86_64 0:7.3.1-15.amzn2                                                   
2022-11-23T01:17:43.9577054Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2022-11-23T01:17:43.9577464Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2022-11-23T01:17:43.9578214Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2022-11-23T01:17:43.9578709Z   emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                    
2022-11-23T01:17:43.9579120Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2022-11-23T01:17:43.9579552Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2022-11-23T01:17:43.9580003Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2022-11-23T01:17:43.9580408Z   glibc-devel.x86_64 0:2.26-62.amzn2                                            
2022-11-23T01:17:43.9580826Z   glibc-headers.x86_64 0:2.26-62.amzn2                                          
2022-11-23T01:17:43.9581241Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2022-11-23T01:17:43.9581650Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2                                   
2022-11-23T01:17:43.9582095Z   kernel-devel.x86_64 0:4.14.296-222.539.amzn2                                  
2022-11-23T01:17:43.9582532Z   kernel-headers.x86_64 0:4.14.296-222.539.amzn2                                
2022-11-23T01:17:43.9582961Z   libatomic.x86_64 0:7.3.1-15.amzn2                                             
2022-11-23T01:17:43.9583353Z   libcilkrts.x86_64 0:7.3.1-15.amzn2                                            
2022-11-23T01:17:43.9583760Z   libgfortran.x86_64 0:7.3.1-15.amzn2                                           
2022-11-23T01:17:43.9584169Z   libitm.x86_64 0:7.3.1-15.amzn2                                                
2022-11-23T01:17:43.9584557Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2022-11-23T01:17:43.9584967Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2022-11-23T01:17:43.9585369Z   libmpx.x86_64 0:7.3.1-15.amzn2                                                
2022-11-23T01:17:43.9585771Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2022-11-23T01:17:43.9586263Z   libquadmath.x86_64 0:7.3.1-15.amzn2                                           
2022-11-23T01:17:43.9586755Z   libsanitizer.x86_64 0:7.3.1-15.amzn2                                          
2022-11-23T01:17:43.9587162Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2022-11-23T01:17:43.9587541Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2022-11-23T01:17:43.9587942Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2022-11-23T01:17:43.9588342Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2022-11-23T01:17:43.9588750Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2022-11-23T01:17:43.9589163Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2022-11-23T01:17:43.9589609Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2022-11-23T01:17:43.9590072Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2022-11-23T01:17:43.9590513Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2022-11-23T01:17:43.9590969Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2022-11-23T01:17:43.9591417Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2022-11-23T01:17:43.9591855Z   systemtap-client.x86_64 0:4.5-1.amzn2.0.1                                     
2022-11-23T01:17:43.9592276Z   systemtap-devel.x86_64 0:4.5-1.amzn2.0.1                                      
2022-11-23T01:17:43.9592699Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2022-11-23T01:17:43.9593108Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2                                        
2022-11-23T01:17:43.9593306Z 
2022-11-23T01:17:43.9593394Z Complete!
2022-11-23T01:17:43.9943806Z ++ uname -r
2022-11-23T01:17:43.9952962Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-11-23T01:17:44.4987798Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:17:44.7791004Z Resolving Dependencies
2022-11-23T01:17:44.7797264Z --> Running transaction check
2022-11-23T01:17:44.7797744Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2022-11-23T01:17:45.0619288Z --> Finished Dependency Resolution
2022-11-23T01:17:45.1415738Z 
2022-11-23T01:17:45.1415982Z Dependencies Resolved
2022-11-23T01:17:45.1422179Z 
2022-11-23T01:17:45.1422635Z ================================================================================
2022-11-23T01:17:45.1423025Z  Package          Arch       Version                       Repository      Size
2022-11-23T01:17:45.1423385Z ================================================================================
2022-11-23T01:17:45.1423638Z Installing:
2022-11-23T01:17:45.1424106Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2022-11-23T01:17:45.1424324Z 
2022-11-23T01:17:45.1424456Z Transaction Summary
2022-11-23T01:17:45.1424750Z ================================================================================
2022-11-23T01:17:45.1425005Z Install  1 Package
2022-11-23T01:17:45.1425164Z 
2022-11-23T01:17:45.1425296Z Total download size: 13 M
2022-11-23T01:17:45.1425559Z Installed size: 60 M
2022-11-23T01:17:45.1425810Z Downloading packages:
2022-11-23T01:17:45.1434318Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-11-23T01:17:45.4363295Z Running transaction check
2022-11-23T01:17:45.4545178Z Running transaction test
2022-11-23T01:17:45.8603160Z Transaction test succeeded
2022-11-23T01:17:45.8606130Z Running transaction
2022-11-23T01:18:00.9403110Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-11-23T01:18:01.0239277Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-11-23T01:18:01.0239556Z 
2022-11-23T01:18:01.0239666Z Installed:
2022-11-23T01:18:01.0240251Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2022-11-23T01:18:01.0240542Z 
2022-11-23T01:18:01.0240659Z Complete!
2022-11-23T01:18:01.0551154Z + sudo modprobe backlight
2022-11-23T01:18:01.0820039Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-515.76.run
2022-11-23T01:18:04.7643828Z + set +e
2022-11-23T01:18:04.7644378Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-11-23T01:18:06.1393882Z Verifying archive integrity... OK
2022-11-23T01:18:32.9769649Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 515.76................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-11-23T01:18:33.1174751Z 
2022-11-23T01:18:33.1175688Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-11-23T01:18:33.1176052Z 
2022-11-23T01:18:46.5196833Z 
2022-11-23T01:18:46.5198080Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-11-23T01:18:46.5201402Z 
2022-11-23T01:18:55.5592920Z + NVIDIA_INSTALLATION_STATUS=0
2022-11-23T01:18:55.5593251Z + RESET_GPU=0
2022-11-23T01:18:55.5593649Z + '[' 0 -ne 0 ']'
2022-11-23T01:18:55.5595928Z ++ command -v nvidia-smi
2022-11-23T01:18:55.5598043Z + '[' -x /usr/bin/nvidia-smi ']'
2022-11-23T01:18:55.5601536Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2022-11-23T01:19:00.6192154Z + INSTALLED_DRIVER_VERSION=515.76
2022-11-23T01:19:00.6192478Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:19:00.6192890Z + '[' 0 -ne 0 ']'
2022-11-23T01:19:00.6193162Z + '[' 0 -eq 1 ']'
2022-11-23T01:19:00.6193460Z + sudo rm -fv /tmp/nvidia_driver
2022-11-23T01:19:00.6805008Z removed ‘/tmp/nvidia_driver’
2022-11-23T01:19:00.6820003Z + set -e
2022-11-23T01:19:00.6820265Z + sudo modprobe nvidia
2022-11-23T01:19:00.6937735Z + echo 'After installing NVIDIA driver'
2022-11-23T01:19:00.6938308Z + lspci
2022-11-23T01:19:00.6938563Z After installing NVIDIA driver
2022-11-23T01:19:00.7113700Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:19:00.7114134Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:19:00.7114539Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:19:00.7114928Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:19:00.7115271Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:19:00.7115655Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:19:00.7119505Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:19:00.7119976Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:19:00.7120591Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:19:00.7121018Z + lsmod
2022-11-23T01:19:00.7135309Z Module                  Size  Used by
2022-11-23T01:19:00.7135861Z nvidia              40808448  0
2022-11-23T01:19:00.7136441Z drm                   425984  1 nvidia
2022-11-23T01:19:00.7136908Z i2c_core               77824  2 nvidia,drm
2022-11-23T01:19:00.7137185Z backlight              16384  0
2022-11-23T01:19:00.7137454Z xt_conntrack           16384  1
2022-11-23T01:19:00.7137728Z ipt_MASQUERADE         16384  1
2022-11-23T01:19:00.7138475Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:19:00.7139418Z nf_conntrack_netlink    49152  0
2022-11-23T01:19:00.7139973Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:19:00.7140499Z xfrm_user              45056  1
2022-11-23T01:19:00.7141054Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:19:00.7141654Z xt_addrtype            16384  2
2022-11-23T01:19:00.7142059Z iptable_filter         16384  1
2022-11-23T01:19:00.7142315Z iptable_nat            16384  1
2022-11-23T01:19:00.7142590Z nf_conntrack_ipv4      16384  3
2022-11-23T01:19:00.7142884Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:19:00.7143166Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:19:00.7143487Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:19:00.7143951Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:19:00.7144337Z br_netfilter           24576  0
2022-11-23T01:19:00.7144614Z bridge                172032  1 br_netfilter
2022-11-23T01:19:00.7144883Z stp                    16384  1 bridge
2022-11-23T01:19:00.7145135Z llc                    16384  2 bridge,stp
2022-11-23T01:19:00.7145395Z overlay                86016  0
2022-11-23T01:19:00.7145651Z sunrpc                393216  1
2022-11-23T01:19:00.7145898Z dm_mirror              28672  0
2022-11-23T01:19:00.7146162Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:19:00.7146465Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:19:00.7146769Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:19:00.7147023Z dax                    69632  1 dm_mod
2022-11-23T01:19:00.7147468Z sb_edac                24576  0
2022-11-23T01:19:00.7148031Z crc32_pclmul           16384  0
2022-11-23T01:19:00.7148564Z ghash_clmulni_intel    16384  0
2022-11-23T01:19:00.7148869Z pcbc                   16384  0
2022-11-23T01:19:00.7149124Z aesni_intel           188416  0
2022-11-23T01:19:00.7149474Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:19:00.7149972Z ata_piix               36864  0
2022-11-23T01:19:00.7150508Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:19:00.7151035Z glue_helper            16384  1 aesni_intel
2022-11-23T01:19:00.7151567Z pcc_cpufreq            16384  0
2022-11-23T01:19:00.7152166Z libata                266240  1 ata_piix
2022-11-23T01:19:00.7152666Z mousedev               24576  0
2022-11-23T01:19:00.7153011Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:19:00.7153479Z evdev                  20480  3
2022-11-23T01:19:00.7153959Z scsi_mod              245760  1 libata
2022-11-23T01:19:00.7154524Z psmouse                32768  0
2022-11-23T01:19:00.7155036Z button                 16384  0
2022-11-23T01:19:00.7155290Z ena                   114688  0
2022-11-23T01:19:00.7155523Z xen_blkfront           49152  2
2022-11-23T01:19:00.7155774Z crc32c_intel           24576  0
2022-11-23T01:19:00.7156025Z autofs4                49152  2
2022-11-23T01:19:00.7156286Z + modinfo nvidia
2022-11-23T01:19:00.7156771Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2022-11-23T01:19:00.7157113Z firmware:       nvidia/515.76/gsp.bin
2022-11-23T01:19:00.7157450Z alias:          char-major-195-*
2022-11-23T01:19:00.7157701Z version:        515.76
2022-11-23T01:19:00.7157955Z supported:      external
2022-11-23T01:19:00.7158394Z license:        NVIDIA
2022-11-23T01:19:00.7158739Z srcversion:     51FD9DD90150B35351AFFBB
2022-11-23T01:19:00.7159050Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2022-11-23T01:19:00.7159360Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2022-11-23T01:19:00.7159649Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2022-11-23T01:19:00.7159983Z depends:        i2c-core,drm
2022-11-23T01:19:00.7160248Z retpoline:      Y
2022-11-23T01:19:00.7160471Z name:           nvidia
2022-11-23T01:19:00.7160870Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2022-11-23T01:19:00.7161243Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2022-11-23T01:19:00.7161616Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2022-11-23T01:19:00.7161981Z parm:           NVreg_ResmanDebugLevel:int
2022-11-23T01:19:00.7162278Z parm:           NVreg_RmLogonRC:int
2022-11-23T01:19:00.7162564Z parm:           NVreg_ModifyDeviceFiles:int
2022-11-23T01:19:00.7162878Z parm:           NVreg_DeviceFileUID:int
2022-11-23T01:19:00.7163176Z parm:           NVreg_DeviceFileGID:int
2022-11-23T01:19:00.7163477Z parm:           NVreg_DeviceFileMode:int
2022-11-23T01:19:00.7163820Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2022-11-23T01:19:00.7164197Z parm:           NVreg_UsePageAttributeTable:int
2022-11-23T01:19:00.7164515Z parm:           NVreg_EnablePCIeGen3:int
2022-11-23T01:19:00.7164791Z parm:           NVreg_EnableMSI:int
2022-11-23T01:19:00.7165079Z parm:           NVreg_TCEBypassMode:int
2022-11-23T01:19:00.7165391Z parm:           NVreg_EnableStreamMemOPs:int
2022-11-23T01:19:00.7165729Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2022-11-23T01:19:00.7166116Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2022-11-23T01:19:00.7166488Z parm:           NVreg_EnableS0ixPowerManagement:int
2022-11-23T01:19:00.7166872Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2022-11-23T01:19:00.7167266Z parm:           NVreg_DynamicPowerManagement:int
2022-11-23T01:19:00.7167680Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2022-11-23T01:19:00.7168057Z parm:           NVreg_EnableGpuFirmware:int
2022-11-23T01:19:00.7168385Z parm:           NVreg_EnableGpuFirmwareLogs:int
2022-11-23T01:19:00.7168744Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2022-11-23T01:19:00.7169103Z parm:           NVreg_EnableUserNUMAManagement:int
2022-11-23T01:19:00.7169413Z parm:           NVreg_MemoryPoolSize:int
2022-11-23T01:19:00.7169729Z parm:           NVreg_KMallocHeapMaxSize:int
2022-11-23T01:19:00.7170049Z parm:           NVreg_VMallocHeapMaxSize:int
2022-11-23T01:19:00.7170341Z parm:           NVreg_IgnoreMMIOCheck:int
2022-11-23T01:19:00.7170642Z parm:           NVreg_NvLinkDisable:int
2022-11-23T01:19:00.7170987Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2022-11-23T01:19:00.7171321Z parm:           NVreg_RegisterPCIDriver:int
2022-11-23T01:19:00.7171644Z parm:           NVreg_EnableDbgBreakpoint:int
2022-11-23T01:19:00.7171968Z parm:           NVreg_RegistryDwords:charp
2022-11-23T01:19:00.7172286Z parm:           NVreg_RegistryDwordsPerDevice:charp
2022-11-23T01:19:00.7172600Z parm:           NVreg_RmMsg:charp
2022-11-23T01:19:00.7172891Z parm:           NVreg_GpuBlacklist:charp
2022-11-23T01:19:00.7173191Z parm:           NVreg_TemporaryFilePath:charp
2022-11-23T01:19:00.7173505Z parm:           NVreg_ExcludedGpus:charp
2022-11-23T01:19:00.7173813Z parm:           NVreg_DmaRemapPeerMmio:int
2022-11-23T01:19:00.7174121Z parm:           rm_firmware_active:charp
2022-11-23T01:19:00.7174361Z + set +e
2022-11-23T01:19:00.7174625Z + nvidia-smi
2022-11-23T01:19:04.5014617Z Wed Nov 23 01:19:04 2022       
2022-11-23T01:19:04.5015809Z +-----------------------------------------------------------------------------+
2022-11-23T01:19:04.5016365Z | NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
2022-11-23T01:19:04.5017069Z |-------------------------------+----------------------+----------------------+
2022-11-23T01:19:04.5017676Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-11-23T01:19:04.5018555Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-11-23T01:19:04.5018916Z |                               |                      |               MIG M. |
2022-11-23T01:19:04.5019195Z |===============================+======================+======================|
2022-11-23T01:19:04.5080627Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |           6281292990 |
2022-11-23T01:19:04.5081357Z | N/A   28C    P0    37W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-11-23T01:19:04.5081975Z |                               |                      |                  N/A |
2022-11-23T01:19:04.5082419Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:19:04.5128585Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
2022-11-23T01:19:04.5129246Z | N/A   23C    P0    38W / 150W |      0MiB /  7680MiB |     99%      Default |
2022-11-23T01:19:04.5129954Z |                               |                      |                  N/A |
2022-11-23T01:19:04.5130524Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:19:04.5130876Z                                                                                
2022-11-23T01:19:04.5131299Z +-----------------------------------------------------------------------------+
2022-11-23T01:19:04.5131648Z | Processes:                                                                  |
2022-11-23T01:19:04.5131991Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-11-23T01:19:04.5132323Z |        ID   ID                                                   Usage      |
2022-11-23T01:19:04.5132623Z |=============================================================================|
2022-11-23T01:19:04.5154811Z |  No running processes found                                                 |
2022-11-23T01:19:04.5155876Z +-----------------------------------------------------------------------------+
2022-11-23T01:19:05.0424665Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:19:05.0425340Z + '[' 0 -eq 0 ']'
2022-11-23T01:19:05.0426026Z + echo 'INFO: Ignoring allowed status 0'
2022-11-23T01:19:05.0426323Z + set -e
2022-11-23T01:19:05.0426574Z INFO: Ignoring allowed status 0
2022-11-23T01:19:05.0429994Z == Installing nvidia container toolkit for amzn2 ==
2022-11-23T01:19:05.0434087Z + sudo yum install -y yum-utils
2022-11-23T01:19:05.5852161Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:19:05.8566996Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-11-23T01:19:05.8567579Z Nothing to do
2022-11-23T01:19:05.8763691Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:19:06.4020708Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:19:06.4321827Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:19:06.4323113Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:19:06.4323652Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:19:06.4463767Z + sudo yum install -y nvidia-docker2
2022-11-23T01:19:06.9686513Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:19:07.0134553Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2022-11-23T01:19:07.0257777Z Importing GPG key 0xF796ECB0:
2022-11-23T01:19:07.0258176Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-11-23T01:19:07.0258863Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-11-23T01:19:07.0259368Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2022-11-23T01:19:07.4192963Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-11-23T01:19:07.4275189Z Importing GPG key 0xF796ECB0:
2022-11-23T01:19:07.4275779Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-11-23T01:19:07.4276208Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-11-23T01:19:07.4276733Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-11-23T01:19:07.6511687Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2022-11-23T01:19:07.6584523Z Importing GPG key 0xF796ECB0:
2022-11-23T01:19:07.6584941Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-11-23T01:19:07.6585318Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-11-23T01:19:07.6585782Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2022-11-23T01:19:09.3687414Z Resolving Dependencies
2022-11-23T01:19:09.3694162Z --> Running transaction check
2022-11-23T01:19:09.3694653Z ---> Package nvidia-docker2.noarch 0:2.11.0-1 will be installed
2022-11-23T01:19:09.3720333Z --> Processing Dependency: nvidia-container-toolkit >= 1.10.0-1 for package: nvidia-docker2-2.11.0-1.noarch
2022-11-23T01:19:09.4097303Z --> Running transaction check
2022-11-23T01:19:09.4097760Z ---> Package nvidia-container-toolkit.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:19:09.4251851Z --> Processing Dependency: nvidia-container-toolkit-base = 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-11-23T01:19:09.4262632Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-11-23T01:19:09.4393103Z --> Processing Dependency: libnvidia-container-tools >= 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-11-23T01:19:09.4393589Z --> Running transaction check
2022-11-23T01:19:09.4394552Z ---> Package libnvidia-container-tools.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:19:09.4404684Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.11.0-1 for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-11-23T01:19:09.4432497Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-11-23T01:19:09.4433210Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-11-23T01:19:09.4433997Z ---> Package nvidia-container-toolkit-base.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:19:09.4436510Z --> Running transaction check
2022-11-23T01:19:09.4437106Z ---> Package libnvidia-container1.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:19:09.7328406Z --> Finished Dependency Resolution
2022-11-23T01:19:09.8071010Z 
2022-11-23T01:19:09.8071350Z Dependencies Resolved
2022-11-23T01:19:09.8084427Z 
2022-11-23T01:19:09.8084678Z ================================================================================
2022-11-23T01:19:09.8085322Z  Package                        Arch    Version      Repository            Size
2022-11-23T01:19:09.8085825Z ================================================================================
2022-11-23T01:19:09.8086101Z Installing:
2022-11-23T01:19:09.8086554Z  nvidia-docker2                 noarch  2.11.0-1     libnvidia-container  8.7 k
2022-11-23T01:19:09.8086900Z Installing for dependencies:
2022-11-23T01:19:09.8087357Z  libnvidia-container-tools      x86_64  1.11.0-1     libnvidia-container   49 k
2022-11-23T01:19:09.8087849Z  libnvidia-container1           x86_64  1.11.0-1     libnvidia-container  1.0 M
2022-11-23T01:19:09.8088359Z  nvidia-container-toolkit       x86_64  1.11.0-1     libnvidia-container  780 k
2022-11-23T01:19:09.8088877Z  nvidia-container-toolkit-base  x86_64  1.11.0-1     libnvidia-container  2.5 M
2022-11-23T01:19:09.8089137Z 
2022-11-23T01:19:09.8089256Z Transaction Summary
2022-11-23T01:19:09.8089521Z ================================================================================
2022-11-23T01:19:09.8089997Z Install  1 Package (+4 Dependent packages)
2022-11-23T01:19:09.8090300Z 
2022-11-23T01:19:09.8090435Z Total download size: 4.3 M
2022-11-23T01:19:09.8090679Z Installed size: 12 M
2022-11-23T01:19:09.8090940Z Downloading packages:
2022-11-23T01:19:09.9303533Z --------------------------------------------------------------------------------
2022-11-23T01:19:09.9304307Z Total                                               35 MB/s | 4.3 MB  00:00     
2022-11-23T01:19:09.9348869Z Running transaction check
2022-11-23T01:19:09.9519964Z Running transaction test
2022-11-23T01:19:09.9682341Z Transaction test succeeded
2022-11-23T01:19:09.9693537Z Running transaction
2022-11-23T01:19:10.4725252Z   Installing : nvidia-container-toolkit-base-1.11.0-1.x86_64                1/5 
2022-11-23T01:19:10.5302335Z   Installing : libnvidia-container1-1.11.0-1.x86_64                         2/5 
2022-11-23T01:19:10.6475234Z   Installing : libnvidia-container-tools-1.11.0-1.x86_64                    3/5 
2022-11-23T01:19:10.7185931Z   Installing : nvidia-container-toolkit-1.11.0-1.x86_64                     4/5 
2022-11-23T01:19:10.7607588Z   Installing : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-11-23T01:19:10.7783312Z   Verifying  : libnvidia-container1-1.11.0-1.x86_64                         1/5 
2022-11-23T01:19:10.7934160Z   Verifying  : nvidia-container-toolkit-base-1.11.0-1.x86_64                2/5 
2022-11-23T01:19:10.8091131Z   Verifying  : nvidia-container-toolkit-1.11.0-1.x86_64                     3/5 
2022-11-23T01:19:10.8242432Z   Verifying  : libnvidia-container-tools-1.11.0-1.x86_64                    4/5 
2022-11-23T01:19:10.9116644Z   Verifying  : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-11-23T01:19:10.9116924Z 
2022-11-23T01:19:10.9117036Z Installed:
2022-11-23T01:19:10.9117430Z   nvidia-docker2.noarch 0:2.11.0-1                                              
2022-11-23T01:19:10.9117639Z 
2022-11-23T01:19:10.9117753Z Dependency Installed:
2022-11-23T01:19:10.9118200Z   libnvidia-container-tools.x86_64 0:1.11.0-1                                   
2022-11-23T01:19:10.9118678Z   libnvidia-container1.x86_64 0:1.11.0-1                                        
2022-11-23T01:19:10.9119124Z   nvidia-container-toolkit.x86_64 0:1.11.0-1                                    
2022-11-23T01:19:10.9119609Z   nvidia-container-toolkit-base.x86_64 0:1.11.0-1                               
2022-11-23T01:19:10.9119841Z 
2022-11-23T01:19:10.9119945Z Complete!
2022-11-23T01:19:11.0112943Z + sudo systemctl restart docker
2022-11-23T01:19:18.8040139Z Command completed after 1 attempt(s).
2022-11-23T01:19:18.8040383Z 
2022-11-23T01:19:18.8042890Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:19:18.8089076Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-11-23T01:19:18.8089476Z python3 -m pip install psutil==5.9.1
2022-11-23T01:19:18.8089800Z python3 -m pip install pynvml==11.4.1
2022-11-23T01:19:18.8090156Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-11-23T01:19:18.8090542Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2022-11-23T01:19:18.8104003Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:19:18.8104308Z env:
2022-11-23T01:19:18.8104553Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:18.8104807Z   GPU_FLAG: --gpus all
2022-11-23T01:19:18.8105058Z ##[endgroup]
2022-11-23T01:19:19.8055751Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:19:20.1854961Z Collecting psutil==5.9.1
2022-11-23T01:19:20.2039078Z   Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2022-11-23T01:19:20.2759588Z Installing collected packages: psutil
2022-11-23T01:19:20.4310545Z Successfully installed psutil-5.9.1
2022-11-23T01:19:20.8997182Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:19:20.9675544Z Collecting pynvml==11.4.1
2022-11-23T01:19:20.9832400Z   Downloading pynvml-11.4.1-py3-none-any.whl (46 kB)
2022-11-23T01:19:21.0318315Z Installing collected packages: pynvml
2022-11-23T01:19:21.0855546Z Successfully installed pynvml-11.4.1
2022-11-23T01:19:21.1359856Z Prepare all required actions
2022-11-23T01:19:21.1360221Z Getting action download info
2022-11-23T01:19:21.2953885Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2022-11-23T01:19:21.5436852Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7)
2022-11-23T01:19:21.7418721Z ##[group]Run ./.github/actions/download-build-artifacts
2022-11-23T01:19:21.7440177Z with:
2022-11-23T01:19:21.7440456Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:19:21.7440717Z env:
2022-11-23T01:19:21.7440937Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:21.7441188Z   GPU_FLAG: --gpus all
2022-11-23T01:19:21.7441425Z ##[endgroup]
2022-11-23T01:19:21.7469599Z ##[group]Run seemethere/download-artifact-s3@v4
2022-11-23T01:19:21.7469889Z with:
2022-11-23T01:19:21.7470150Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:19:21.7470456Z   s3-bucket: gha-artifacts
2022-11-23T01:19:21.7470738Z   region: us-east-1
2022-11-23T01:19:21.7470967Z env:
2022-11-23T01:19:21.7471202Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:21.7471446Z   GPU_FLAG: --gpus all
2022-11-23T01:19:21.7471689Z ##[endgroup]
2022-11-23T01:19:22.8373345Z Found 1 objects with prefix pytorch/pytorch/3528293562/linux-bionic-cuda11.6-py3.10-gcc7/
2022-11-23T01:19:22.8373936Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:19:37.2264154Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:19:37.2264668Z 
2022-11-23T01:19:37.2269935Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:19:37.2271252Z Artifact download has finished successfully
2022-11-23T01:19:37.2547372Z ##[group]Run unzip -o artifacts.zip
2022-11-23T01:19:37.2547703Z unzip -o artifacts.zip
2022-11-23T01:19:37.2561138Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:19:37.2561420Z env:
2022-11-23T01:19:37.2561663Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:37.2561932Z   GPU_FLAG: --gpus all
2022-11-23T01:19:37.2562167Z ##[endgroup]
2022-11-23T01:19:37.2669271Z Archive:  artifacts.zip
2022-11-23T01:19:37.2671184Z    creating: dist/
2022-11-23T01:19:39.3397563Z   inflating: dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl  
2022-11-23T01:19:39.3398085Z    creating: build/custom_test_artifacts/
2022-11-23T01:19:39.3398511Z    creating: build/custom_test_artifacts/custom-op-build/
2022-11-23T01:19:39.3399004Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-11-23T01:19:39.3405556Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:19:39.3406144Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-11-23T01:19:39.3406710Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:19:39.3407273Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:19:39.3407804Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:19:39.3410320Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:19:39.3411512Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:19:39.3412073Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:19:39.3412872Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:19:39.3415446Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:19:39.3416629Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:19:39.3418401Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:19:39.3419361Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:19:39.3420826Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:19:39.3421969Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:19:39.3422562Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:19:39.3423113Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:19:39.3477317Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:19:39.3478040Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:19:39.3478756Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:19:39.3479485Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:19:39.3480191Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:19:39.3480891Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:19:39.3481581Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:19:39.3482271Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:19:39.3483155Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:19:39.3525493Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:19:39.3566711Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:19:39.3567778Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:19:39.3568515Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:19:39.3569524Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:19:39.3570506Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:19:39.3571150Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:19:39.3572041Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:19:39.3574015Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:19:39.3647009Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:19:39.3720001Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:19:39.3720907Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:19:39.3721562Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-11-23T01:19:39.3722459Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-11-23T01:19:39.3723020Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:19:39.3723553Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-11-23T01:19:39.3724130Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-11-23T01:19:39.3724744Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-11-23T01:19:39.3725325Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-11-23T01:19:39.3725906Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-11-23T01:19:39.3726488Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:19:39.3727559Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-11-23T01:19:39.3728336Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-11-23T01:19:39.3728955Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-11-23T01:19:39.3729549Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-11-23T01:19:39.3750860Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-11-23T01:19:39.3864949Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-11-23T01:19:39.3865527Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-11-23T01:19:39.3866130Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-11-23T01:19:39.3866756Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-11-23T01:19:39.3867369Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-11-23T01:19:39.3867958Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-11-23T01:19:39.3868564Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:19:39.3869382Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-11-23T01:19:39.3869996Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-11-23T01:19:39.3870602Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-11-23T01:19:39.3871201Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-11-23T01:19:39.3892124Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-11-23T01:19:39.3975157Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-11-23T01:19:39.3975805Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:19:39.3976412Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:19:39.3976974Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-11-23T01:19:39.3977762Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-11-23T01:19:39.3979397Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:19:39.3979933Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-11-23T01:19:39.3983631Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-11-23T01:19:39.3984556Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-11-23T01:19:39.3985403Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-11-23T01:19:39.4078055Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-11-23T01:19:39.4141307Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-11-23T01:19:39.4141799Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-11-23T01:19:39.4142263Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-11-23T01:19:39.4148905Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:19:39.4149429Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-11-23T01:19:39.4149972Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:19:39.4150532Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:19:39.4151084Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:19:39.4153309Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:19:39.4154399Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:19:39.4154962Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:19:39.4155519Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:19:39.4158213Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:19:39.4159291Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:19:39.4161213Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:19:39.4161831Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:19:39.4163247Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:19:39.4164386Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:19:39.4164964Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:19:39.4165525Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:19:39.4220056Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:19:39.4220782Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:19:39.4221502Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:19:39.4222231Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:19:39.4222942Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:19:39.4223611Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:19:39.4224300Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:19:39.4225172Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:19:39.4226170Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:19:39.4268353Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:19:39.4309447Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:19:39.4310490Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:19:39.4311228Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:19:39.4312065Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:19:39.4312810Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:19:39.4314012Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:19:39.4315005Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:19:39.4317001Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:19:39.4390026Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:19:39.4463124Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:19:39.4463757Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:19:39.4464313Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-11-23T01:19:39.4465082Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-11-23T01:19:39.4465768Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:19:39.4466312Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-11-23T01:19:39.4466895Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-11-23T01:19:39.4467508Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-11-23T01:19:39.4468085Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-11-23T01:19:39.4468857Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-11-23T01:19:39.4469471Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-11-23T01:19:39.4470603Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-11-23T01:19:39.4471395Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-11-23T01:19:39.4472167Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-11-23T01:19:39.4472765Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-11-23T01:19:39.4493579Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-11-23T01:19:39.4557314Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-11-23T01:19:39.4557945Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:19:39.4558685Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:19:39.4559226Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-11-23T01:19:39.4560088Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-11-23T01:19:39.4561103Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:19:39.4561633Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-11-23T01:19:39.4564547Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-11-23T01:19:39.4565083Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-11-23T01:19:39.4566048Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-11-23T01:19:39.4615309Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-11-23T01:19:39.4615783Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-11-23T01:19:39.4616273Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-11-23T01:19:39.4623370Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:19:39.4623943Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-11-23T01:19:39.4624504Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:19:39.4625091Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:19:39.4625661Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:19:39.4628212Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:19:39.4629317Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:19:39.4629923Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:19:39.4630510Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:19:39.4633239Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:19:39.4634312Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:19:39.4636258Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:19:39.4636904Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:19:39.4638268Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:19:39.4639372Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:19:39.4639972Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:19:39.4640564Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:19:39.4694739Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:19:39.4695525Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:19:39.4696255Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:19:39.4697714Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:19:39.4698919Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:19:39.4699639Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:19:39.4700432Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:19:39.4701144Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:19:39.4702057Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:19:39.4744459Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:19:39.4785725Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:19:39.4786732Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:19:39.4787474Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:19:39.4788312Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:19:39.4789109Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:19:39.4790094Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:19:39.4791096Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:19:39.4793171Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:19:39.4866253Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:19:39.4939195Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:19:39.4940044Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:19:39.4940655Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-11-23T01:19:39.4941396Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-11-23T01:19:39.4941981Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:19:39.4942536Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-11-23T01:19:39.4943156Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-11-23T01:19:39.4943812Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-11-23T01:19:39.4944449Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-11-23T01:19:39.4945067Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-11-23T01:19:39.4945676Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:19:39.4946847Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-11-23T01:19:39.4947607Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-11-23T01:19:39.4948232Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-11-23T01:19:39.4948988Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-11-23T01:19:39.4953723Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-11-23T01:19:39.5101138Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-11-23T01:19:39.5101765Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-11-23T01:19:39.5102402Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-11-23T01:19:39.5103065Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-11-23T01:19:39.5103904Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-11-23T01:19:39.5104530Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-11-23T01:19:39.5105286Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:19:39.5106403Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-11-23T01:19:39.5107198Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-11-23T01:19:39.5107850Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-11-23T01:19:39.5108497Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-11-23T01:19:39.5129680Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-11-23T01:19:39.5188231Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-11-23T01:19:39.5188915Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:19:39.5189549Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:19:39.5190133Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-11-23T01:19:39.5190842Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-11-23T01:19:39.5192036Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:19:39.5192596Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-11-23T01:19:39.5195633Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-11-23T01:19:39.5196319Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-11-23T01:19:39.5197095Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-11-23T01:19:39.5316626Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-11-23T01:19:39.5362118Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-11-23T01:19:39.5362492Z    creating: build/lib/
2022-11-23T01:19:39.5428153Z   inflating: build/lib/libgtest.a    
2022-11-23T01:19:39.5429248Z   inflating: build/lib/libclog.a     
2022-11-23T01:19:39.5439780Z   inflating: build/lib/libpthreadpool.a  
2022-11-23T01:19:39.5546118Z   inflating: build/lib/libprotobuf-lite.a  
2022-11-23T01:19:39.5555255Z   inflating: build/lib/libittnotify.a  
2022-11-23T01:19:39.5646548Z   inflating: build/lib/libbenchmark.a  
2022-11-23T01:19:39.5678605Z   inflating: build/lib/libtensorpipe_uv.a  
2022-11-23T01:19:39.5755111Z   inflating: build/lib/libasmjit.a   
2022-11-23T01:19:39.6288547Z   inflating: build/lib/libprotobuf.a  
2022-11-23T01:19:39.6421614Z   inflating: build/lib/libgloo.a     
2022-11-23T01:19:39.6453745Z   inflating: build/lib/libfmt.a      
2022-11-23T01:19:39.6455562Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-11-23T01:19:39.6456067Z   inflating: build/lib/libfoxi_loader.a  
2022-11-23T01:19:39.6535876Z   inflating: build/lib/libc10.so     
2022-11-23T01:19:39.6537032Z   inflating: build/lib/libtorch_global_deps.so  
2022-11-23T01:19:39.6547305Z   inflating: build/lib/libcpuinfo.a  
2022-11-23T01:19:39.6556179Z   inflating: build/lib/libcpuinfo_internals.a  
2022-11-23T01:19:39.7126142Z   inflating: build/lib/libprotoc.a   
2022-11-23T01:19:39.7142360Z   inflating: build/lib/libqnnpack.a  
2022-11-23T01:19:39.7144981Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-11-23T01:19:39.7167793Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-11-23T01:19:39.7186839Z   inflating: build/lib/libgmock.a    
2022-11-23T01:19:39.7208932Z   inflating: build/lib/libnnpack.a   
2022-11-23T01:19:39.7209628Z   inflating: build/lib/libgtest_main.a  
2022-11-23T01:19:39.7210592Z   inflating: build/lib/libbenchmark_main.a  
2022-11-23T01:19:40.6992203Z   inflating: build/lib/libdnnl.a     
2022-11-23T01:19:40.7133326Z   inflating: build/lib/libXNNPACK.a  
2022-11-23T01:19:40.7788886Z   inflating: build/lib/libtensorpipe.a  
2022-11-23T01:19:40.7841921Z   inflating: build/lib/libc10_cuda.so  
2022-11-23T01:19:40.9383681Z   inflating: build/lib/libfbgemm.a   
2022-11-23T01:19:40.9384337Z   inflating: build/lib/libgmock_main.a  
2022-11-23T01:19:41.0531716Z   inflating: build/lib/libdnnl_graph.a  
2022-11-23T01:19:41.1047780Z   inflating: build/lib/libkineto.a   
2022-11-23T01:19:41.1337618Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-11-23T01:19:41.1383354Z   inflating: build/lib/libcaffe2_protos.a  
2022-11-23T01:19:41.1431480Z   inflating: build/lib/libonnx_proto.a  
2022-11-23T01:19:41.2110262Z   inflating: build/lib/libonnx.a     
2022-11-23T01:19:41.2545481Z   inflating: build/lib/libgloo_cuda.a  
2022-11-23T01:19:43.6154577Z   inflating: build/lib/libtorch_cpu.so  
2022-11-23T01:19:45.7525549Z   inflating: build/lib/libtorch_cuda.so  
2022-11-23T01:19:45.7526388Z   inflating: build/lib/libtorch.so   
2022-11-23T01:19:46.7411571Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-11-23T01:19:46.7414396Z   inflating: build/lib/libc10d_cuda_test.so  
2022-11-23T01:19:46.7438359Z   inflating: build/lib/libjitbackend_test.so  
2022-11-23T01:19:46.7498333Z   inflating: build/lib/libtorchbind_test.so  
2022-11-23T01:19:46.7529601Z   inflating: build/lib/libbackend_with_compiler.so  
2022-11-23T01:19:46.7534305Z   inflating: build/lib/libshm.so     
2022-11-23T01:19:46.9337341Z   inflating: build/lib/libtorch_python.so  
2022-11-23T01:19:46.9376937Z   inflating: build/lib/libnnapi_backend.so  
2022-11-23T01:19:46.9377256Z    creating: build/bin/
2022-11-23T01:19:46.9430020Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-11-23T01:19:46.9484922Z   inflating: build/bin/c10_DeviceGuard_test  
2022-11-23T01:19:46.9538565Z   inflating: build/bin/c10_Device_test  
2022-11-23T01:19:46.9601531Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-11-23T01:19:46.9652734Z   inflating: build/bin/c10_StreamGuard_test  
2022-11-23T01:19:46.9705353Z   inflating: build/bin/c10_SymInt_test  
2022-11-23T01:19:46.9764755Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-11-23T01:19:46.9824423Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-11-23T01:19:46.9885266Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-11-23T01:19:46.9936425Z   inflating: build/bin/c10_Array_test  
2022-11-23T01:19:46.9993325Z   inflating: build/bin/c10_Bitset_test  
2022-11-23T01:19:47.0048039Z   inflating: build/bin/c10_C++17_test  
2022-11-23T01:19:47.0099610Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-11-23T01:19:47.0151916Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-11-23T01:19:47.0204838Z   inflating: build/bin/c10_Half_test  
2022-11-23T01:19:47.0272067Z   inflating: build/bin/c10_Metaprogramming_test  
2022-11-23T01:19:47.0332480Z   inflating: build/bin/c10_LeftRight_test  
2022-11-23T01:19:47.0488432Z   inflating: build/bin/c10_SmallVectorTest  
2022-11-23T01:19:47.0542022Z   inflating: build/bin/c10_Synchronized_test  
2022-11-23T01:19:47.0603149Z   inflating: build/bin/c10_ThreadLocal_test  
2022-11-23T01:19:47.0659543Z   inflating: build/bin/c10_TypeIndex_test  
2022-11-23T01:19:47.0713174Z   inflating: build/bin/c10_TypeList_test  
2022-11-23T01:19:47.0764348Z   inflating: build/bin/c10_TypeTraits_test  
2022-11-23T01:19:47.0819796Z   inflating: build/bin/c10_accumulate_test  
2022-11-23T01:19:47.0879273Z   inflating: build/bin/c10_bfloat16_test  
2022-11-23T01:19:47.0936767Z   inflating: build/bin/c10_complex_math_test  
2022-11-23T01:19:47.0996166Z   inflating: build/bin/c10_complex_test  
2022-11-23T01:19:47.1113808Z   inflating: build/bin/c10_either_test  
2022-11-23T01:19:47.1169633Z   inflating: build/bin/c10_exception_test  
2022-11-23T01:19:47.1223021Z   inflating: build/bin/c10_flags_test  
2022-11-23T01:19:47.1405855Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-11-23T01:19:47.1459832Z   inflating: build/bin/c10_irange_test  
2022-11-23T01:19:47.1521356Z   inflating: build/bin/c10_logging_test  
2022-11-23T01:19:47.1601217Z   inflating: build/bin/c10_optional_test  
2022-11-23T01:19:47.1667839Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-11-23T01:19:47.1725970Z   inflating: build/bin/c10_registry_test  
2022-11-23T01:19:47.1789279Z   inflating: build/bin/c10_string_view_test  
2022-11-23T01:19:47.1844202Z   inflating: build/bin/c10_tempfile_test  
2022-11-23T01:19:47.1904701Z   inflating: build/bin/c10_typeid_test  
2022-11-23T01:19:47.1964863Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-11-23T01:19:47.2487068Z   inflating: build/bin/protoc-3.13.0.0  
2022-11-23T01:19:47.3009082Z   inflating: build/bin/protoc        
2022-11-23T01:19:47.3060915Z   inflating: build/bin/c10_cuda_CUDATest  
2022-11-23T01:19:47.3378046Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-11-23T01:19:47.3733524Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-11-23T01:19:47.3791168Z   inflating: build/bin/FileStoreTest  
2022-11-23T01:19:47.3848260Z   inflating: build/bin/HashStoreTest  
2022-11-23T01:19:47.3913268Z   inflating: build/bin/TCPStoreTest  
2022-11-23T01:19:47.3929167Z   inflating: build/bin/ProcessGroupMPITest  
2022-11-23T01:19:47.3932485Z   inflating: build/bin/example_allreduce  
2022-11-23T01:19:47.3988511Z   inflating: build/bin/Dimname_test  
2022-11-23T01:19:47.4056496Z   inflating: build/bin/MaybeOwned_test  
2022-11-23T01:19:47.4134933Z   inflating: build/bin/Dict_test     
2022-11-23T01:19:47.4196085Z   inflating: build/bin/NamedTensor_test  
2022-11-23T01:19:47.4259337Z   inflating: build/bin/apply_utils_test  
2022-11-23T01:19:47.4322189Z   inflating: build/bin/atest         
2022-11-23T01:19:47.4379849Z   inflating: build/bin/broadcast_test  
2022-11-23T01:19:47.4442523Z   inflating: build/bin/cpu_generator_test  
2022-11-23T01:19:47.4508408Z   inflating: build/bin/basic         
2022-11-23T01:19:47.4564193Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-11-23T01:19:47.4617281Z   inflating: build/bin/dispatch_key_set_test  
2022-11-23T01:19:47.4670444Z   inflating: build/bin/dlconvertor_test  
2022-11-23T01:19:47.4765527Z   inflating: build/bin/cpu_rng_test  
2022-11-23T01:19:47.4827845Z   inflating: build/bin/extension_backend_test  
2022-11-23T01:19:47.4887475Z   inflating: build/bin/half_test     
2022-11-23T01:19:47.4989187Z   inflating: build/bin/ivalue_test   
2022-11-23T01:19:47.5041677Z   inflating: build/bin/lazy_tensor_test  
2022-11-23T01:19:47.5099177Z   inflating: build/bin/math_kernel_test  
2022-11-23T01:19:47.5156232Z   inflating: build/bin/memory_overlapping_test  
2022-11-23T01:19:47.5213659Z   inflating: build/bin/memory_format_test  
2022-11-23T01:19:47.5269759Z   inflating: build/bin/mobile_memory_cleanup  
2022-11-23T01:19:47.5323712Z   inflating: build/bin/operator_name_test  
2022-11-23T01:19:47.5383969Z   inflating: build/bin/native_test   
2022-11-23T01:19:47.5437405Z   inflating: build/bin/operators_test  
2022-11-23T01:19:47.5493412Z   inflating: build/bin/packedtensoraccessor_test  
2022-11-23T01:19:47.5563477Z   inflating: build/bin/pow_test      
2022-11-23T01:19:47.5624767Z   inflating: build/bin/quantized_test  
2022-11-23T01:19:47.5677626Z   inflating: build/bin/reduce_ops_test  
2022-11-23T01:19:47.5731550Z   inflating: build/bin/reportMemoryUsage_test  
2022-11-23T01:19:47.5792081Z   inflating: build/bin/scalar_tensor_test  
2022-11-23T01:19:47.5853220Z   inflating: build/bin/scalar_test   
2022-11-23T01:19:47.5909205Z   inflating: build/bin/stride_properties_test  
2022-11-23T01:19:47.5993809Z   inflating: build/bin/tensor_iterator_test  
2022-11-23T01:19:47.6053415Z   inflating: build/bin/test_parallel  
2022-11-23T01:19:47.6113184Z   inflating: build/bin/type_ptr_test  
2022-11-23T01:19:47.6115839Z   inflating: build/bin/thread_init_test  
2022-11-23T01:19:47.6168698Z   inflating: build/bin/variant_test  
2022-11-23T01:19:47.6233535Z   inflating: build/bin/type_test     
2022-11-23T01:19:47.6289091Z   inflating: build/bin/undefined_tensor_test  
2022-11-23T01:19:47.6343519Z   inflating: build/bin/weakref_test  
2022-11-23T01:19:47.6344862Z   inflating: build/bin/verify_api_visibility  
2022-11-23T01:19:47.6419911Z   inflating: build/bin/vmap_test     
2022-11-23T01:19:47.6474170Z   inflating: build/bin/wrapdim_test  
2022-11-23T01:19:47.6538028Z   inflating: build/bin/IListRef_test  
2022-11-23T01:19:47.6590754Z   inflating: build/bin/xla_tensor_test  
2022-11-23T01:19:47.6709115Z   inflating: build/bin/List_test     
2022-11-23T01:19:47.6840009Z   inflating: build/bin/kernel_function_legacy_test  
2022-11-23T01:19:47.6944070Z   inflating: build/bin/kernel_function_test  
2022-11-23T01:19:47.7013811Z   inflating: build/bin/KernelFunction_test  
2022-11-23T01:19:47.7152266Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-11-23T01:19:47.7263835Z   inflating: build/bin/kernel_lambda_test  
2022-11-23T01:19:47.7327698Z   inflating: build/bin/kernel_stackbased_test  
2022-11-23T01:19:47.7381659Z   inflating: build/bin/CppSignature_test  
2022-11-23T01:19:47.7484749Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-11-23T01:19:47.7536006Z   inflating: build/bin/op_allowlist_test  
2022-11-23T01:19:47.7596351Z   inflating: build/bin/backend_fallback_test  
2022-11-23T01:19:47.7910269Z   inflating: build/bin/op_registration_test  
2022-11-23T01:19:47.7967378Z   inflating: build/bin/inline_container_test  
2022-11-23T01:19:47.8023080Z   inflating: build/bin/cuda_apply_test  
2022-11-23T01:19:47.8087275Z   inflating: build/bin/cuda_atomic_ops_test  
2022-11-23T01:19:47.8144471Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-11-23T01:19:47.8216552Z   inflating: build/bin/cuda_complex_math_test  
2022-11-23T01:19:47.8269060Z   inflating: build/bin/cuda_device_test  
2022-11-23T01:19:47.8331462Z   inflating: build/bin/cuda_complex_test  
2022-11-23T01:19:47.8394713Z   inflating: build/bin/cuda_cub_test  
2022-11-23T01:19:47.8448105Z   inflating: build/bin/cuda_dlconvertor_test  
2022-11-23T01:19:47.8501827Z   inflating: build/bin/cuda_integer_divider_test  
2022-11-23T01:19:47.8573768Z   inflating: build/bin/cuda_distributions_test  
2022-11-23T01:19:47.8636404Z   inflating: build/bin/cuda_generator_test  
2022-11-23T01:19:47.8688954Z   inflating: build/bin/cuda_half_test  
2022-11-23T01:19:47.8753720Z   inflating: build/bin/cuda_stream_test  
2022-11-23T01:19:47.8805603Z   inflating: build/bin/cuda_optional_test  
2022-11-23T01:19:47.8861930Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-11-23T01:19:47.8916467Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-11-23T01:19:47.8968408Z   inflating: build/bin/cuda_cudnn_test  
2022-11-23T01:19:47.9024283Z   inflating: build/bin/cuda_vectorized_test  
2022-11-23T01:19:47.9041973Z   inflating: build/bin/tutorial_tensorexpr  
2022-11-23T01:19:47.9112092Z   inflating: build/bin/ProcessGroupGlooTest  
2022-11-23T01:19:47.9174116Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-11-23T01:19:47.9239978Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-11-23T01:19:47.9302684Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-11-23T01:19:47.9359202Z   inflating: build/bin/ProcessGroupUCCTest  
2022-11-23T01:19:47.9416948Z   inflating: build/bin/test_dist_autograd  
2022-11-23T01:19:47.9419870Z   inflating: build/bin/parallel_benchmark  
2022-11-23T01:19:47.9493585Z   inflating: build/bin/test_mobile_nnc  
2022-11-23T01:19:47.9568643Z   inflating: build/bin/test_cpp_rpc  
2022-11-23T01:19:47.9580214Z   inflating: build/bin/aot_model_compiler_test  
2022-11-23T01:19:48.0494698Z   inflating: build/bin/test_tensorexpr  
2022-11-23T01:19:48.0500356Z   inflating: build/bin/torch_shm_manager  
2022-11-23T01:19:48.0886812Z   inflating: build/bin/test_lazy     
2022-11-23T01:19:48.2212254Z   inflating: build/bin/test_api      
2022-11-23T01:19:48.3421430Z   inflating: build/bin/test_jit      
2022-11-23T01:19:48.3423490Z   inflating: .pytorch-test-times.json  
2022-11-23T01:19:48.3456779Z ##[group]Run df -H
2022-11-23T01:19:48.3457040Z df -H
2022-11-23T01:19:48.3471021Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:19:48.3471323Z env:
2022-11-23T01:19:48.3471568Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:48.3471832Z   GPU_FLAG: --gpus all
2022-11-23T01:19:48.3472085Z ##[endgroup]
2022-11-23T01:19:48.3511936Z Filesystem      Size  Used Avail Use% Mounted on
2022-11-23T01:19:48.3512614Z devtmpfs        129G     0  129G   0% /dev
2022-11-23T01:19:48.3513242Z tmpfs           129G     0  129G   0% /dev/shm
2022-11-23T01:19:48.3513840Z tmpfs           129G  603k  129G   1% /run
2022-11-23T01:19:48.3514165Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-11-23T01:19:48.3514482Z /dev/xvda1      162G   30G  132G  19% /
2022-11-23T01:19:48.3514754Z tmpfs            26G     0   26G   0% /run/user/0
2022-11-23T01:19:48.3563301Z ##[group]Run .github/scripts/parse_ref.py
2022-11-23T01:19:48.3563656Z .github/scripts/parse_ref.py
2022-11-23T01:19:48.3575262Z shell: /usr/bin/bash -e {0}
2022-11-23T01:19:48.3575510Z env:
2022-11-23T01:19:48.3575748Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:48.3576002Z   GPU_FLAG: --gpus all
2022-11-23T01:19:48.3576250Z ##[endgroup]
2022-11-23T01:19:48.3868690Z ##[group]Run set -x
2022-11-23T01:19:48.3869087Z set -x
2022-11-23T01:19:48.3869317Z 
2022-11-23T01:19:48.3869569Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-11-23T01:19:48.3869922Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-11-23T01:19:48.3870271Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-11-23T01:19:48.3870578Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-11-23T01:19:48.3870851Z else
2022-11-23T01:19:48.3871129Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:19:48.3871401Z fi
2022-11-23T01:19:48.3871603Z 
2022-11-23T01:19:48.3871932Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-11-23T01:19:48.3872247Z 
2022-11-23T01:19:48.3872526Z # sanitize the input commit message and PR body here:
2022-11-23T01:19:48.3872818Z #
2022-11-23T01:19:48.3873196Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-11-23T01:19:48.3873683Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-11-23T01:19:48.3874107Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-11-23T01:19:48.3874420Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-11-23T01:19:48.3874658Z 
2022-11-23T01:19:48.3875011Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-11-23T01:19:48.3875386Z # wreak havoc internally
2022-11-23T01:19:48.3875706Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-11-23T01:19:48.3876142Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-11-23T01:19:48.3876405Z 
2022-11-23T01:19:48.3876717Z # detached container should get cleaned up by teardown_ec2_linux
2022-11-23T01:19:48.3877100Z # TODO: Stop building test binaries as part of the build phase
2022-11-23T01:19:48.3877476Z # Used for GPU_FLAG since that doesn't play nice
2022-11-23T01:19:48.3877806Z # shellcheck disable=SC2086,SC2090
2022-11-23T01:19:48.3878109Z container_name=$(docker run \
2022-11-23T01:19:48.3878368Z   ${GPU_FLAG:-} \
2022-11-23T01:19:48.3878639Z   -e BUILD_ENVIRONMENT \
2022-11-23T01:19:48.3878913Z   -e PR_NUMBER \
2022-11-23T01:19:48.3879161Z   -e GITHUB_ACTIONS \
2022-11-23T01:19:48.3879423Z   -e BASE_SHA \
2022-11-23T01:19:48.3879674Z   -e BRANCH \
2022-11-23T01:19:48.3879899Z   -e SHA1 \
2022-11-23T01:19:48.3880158Z   -e AWS_DEFAULT_REGION \
2022-11-23T01:19:48.3880436Z   -e IN_WHEEL_TEST \
2022-11-23T01:19:48.3880680Z   -e SHARD_NUMBER \
2022-11-23T01:19:48.3880941Z   -e TEST_CONFIG \
2022-11-23T01:19:48.3881207Z   -e NUM_TEST_SHARDS \
2022-11-23T01:19:48.3881449Z   -e PR_BODY \
2022-11-23T01:19:48.3881711Z   -e COMMIT_MESSAGES \
2022-11-23T01:19:48.3881997Z   -e PYTORCH_RETRY_TEST_CASES \
2022-11-23T01:19:48.3882289Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-11-23T01:19:48.3882578Z   -e PR_LABELS \
2022-11-23T01:19:48.3882867Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-11-23T01:19:48.3883158Z   -e SCCACHE_BUCKET \
2022-11-23T01:19:48.3883420Z   -e SCCACHE_S3_KEY_PREFIX \
2022-11-23T01:19:48.3883692Z   -e XLA_CUDA \
2022-11-23T01:19:48.3883978Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-11-23T01:19:48.3884280Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2022-11-23T01:19:48.3884615Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2022-11-23T01:19:48.3884970Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-11-23T01:19:48.3885274Z   --ulimit stack=10485760:83886080 \
2022-11-23T01:19:48.3885587Z   --security-opt seccomp=unconfined \
2022-11-23T01:19:48.3885965Z   --cap-add=SYS_PTRACE \
2022-11-23T01:19:48.3886249Z   --ipc=host \
2022-11-23T01:19:48.3886499Z   --shm-size="${SHM_SIZE}" \
2022-11-23T01:19:48.3886764Z   --tty \
2022-11-23T01:19:48.3887007Z   --detach \
2022-11-23T01:19:48.3887257Z   --name="${container_name}" \
2022-11-23T01:19:48.3887534Z   --user jenkins \
2022-11-23T01:19:48.3887857Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-11-23T01:19:48.3888183Z   -w /var/lib/jenkins/workspace \
2022-11-23T01:19:48.3888468Z   "${DOCKER_IMAGE}"
2022-11-23T01:19:48.3888712Z )
2022-11-23T01:19:48.3889000Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2022-11-23T01:19:48.3889443Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2022-11-23T01:19:48.3902243Z shell: /usr/bin/bash -e {0}
2022-11-23T01:19:48.3902492Z env:
2022-11-23T01:19:48.3902738Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:48.3902991Z   GPU_FLAG: --gpus all
2022-11-23T01:19:48.3903318Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:19:48.3903628Z   PR_NUMBER: 
2022-11-23T01:19:48.3903845Z   BRANCH: master
2022-11-23T01:19:48.3904122Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:19:48.3904437Z   BASE_SHA: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:19:48.3904718Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T01:19:48.3905003Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T01:19:48.3905283Z   TEST_CONFIG: distributed
2022-11-23T01:19:48.3905536Z   SHARD_NUMBER: 2
2022-11-23T01:19:48.3905804Z   NUM_TEST_SHARDS: 3
2022-11-23T01:19:48.3906149Z   PR_BODY: 
2022-11-23T01:19:48.3906455Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-11-23T01:19:48.3906764Z   SCCACHE_S3_KEY_PREFIX: pull
2022-11-23T01:19:48.3907017Z   SHM_SIZE: 2g
2022-11-23T01:19:48.3907502Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:19:48.3907951Z   XLA_CUDA: 
2022-11-23T01:19:48.3908302Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:19:48.3908680Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2022-11-23T01:19:48.3908962Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2022-11-23T01:19:48.3909235Z ##[endgroup]
2022-11-23T01:19:48.3937153Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-11-23T01:19:48.3937667Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *onnx* ]]
2022-11-23T01:19:48.3938431Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:19:48.3941196Z ++ git cherry -v origin/master
2022-11-23T01:19:48.3957550Z + COMMIT_MESSAGES=
2022-11-23T01:19:48.3958063Z + COMMIT_MESSAGES=
2022-11-23T01:19:48.3958335Z + PR_BODY=
2022-11-23T01:19:48.3958595Z + export COMMIT_MESSAGES=
2022-11-23T01:19:48.3958838Z + COMMIT_MESSAGES=
2022-11-23T01:19:48.3959091Z + export PR_BODY=
2022-11-23T01:19:48.3959329Z + PR_BODY=
2022-11-23T01:19:48.3968379Z +++ nproc --ignore=2
2022-11-23T01:19:48.3993941Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528293562 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:20:03.3071079Z + container_name=846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T01:20:03.3071612Z + echo DOCKER_CONTAINER_ID=846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T01:20:03.3076075Z ++ echo dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:20:03.3077741Z + docker exec -t 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e sh -c 'pip install dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2022-11-23T01:20:03.8898796Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:20:04.8465786Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (4.4.0)
2022-11-23T01:20:04.8469057Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3)
2022-11-23T01:20:04.8473869Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (1.11.1)
2022-11-23T01:20:04.8490830Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (3.3.0)
2022-11-23T01:20:04.8572623Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==1.14.0a0+git1cfd385) (1.21.2)
2022-11-23T01:20:04.8797161Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1)
2022-11-23T01:20:05.8253366Z Installing collected packages: torch
2022-11-23T01:20:15.7481282Z Successfully installed torch-1.14.0a0+git1cfd385
2022-11-23T01:20:15.8131873Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-11-23T01:20:15.8351670Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2022-11-23T01:20:15.8352332Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T01:20:15.8352804Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2022-11-23T01:20:15.8355862Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2022-11-23T01:20:15.8356230Z + BUILD_DIR=build
2022-11-23T01:20:15.8356499Z + BUILD_RENAMED_DIR=build_renamed
2022-11-23T01:20:15.8356773Z + BUILD_BIN_DIR=build/bin
2022-11-23T01:20:15.8357017Z + export VALGRIND=ON
2022-11-23T01:20:15.8357263Z + VALGRIND=ON
2022-11-23T01:20:15.8357651Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *clang9* ]]
2022-11-23T01:20:15.8361203Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *bazel* ]]
2022-11-23T01:20:15.8361594Z ++ realpath build/custom_test_artifacts
2022-11-23T01:20:15.8365621Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-11-23T01:20:15.8368826Z ++ dirname .jenkins/pytorch/test.sh
2022-11-23T01:20:15.8376406Z + source .jenkins/pytorch/common.sh
2022-11-23T01:20:15.8380336Z +++ dirname .jenkins/pytorch/common.sh
2022-11-23T01:20:15.8390847Z ++ source .jenkins/pytorch/common_utils.sh
2022-11-23T01:20:15.8392974Z +++ declare -f -t trap_add
2022-11-23T01:20:15.8399218Z ++ set -ex
2022-11-23T01:20:15.8399612Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:20:15.8399920Z ++ BUILD_TEST_LIBTORCH=0
2022-11-23T01:20:15.8400326Z + echo 'Environment variables'
2022-11-23T01:20:15.8400604Z Environment variables
2022-11-23T01:20:15.8400834Z + env
2022-11-23T01:20:15.8409589Z SHARD_NUMBER=2
2022-11-23T01:20:15.8410230Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1
2022-11-23T01:20:15.8410918Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6
2022-11-23T01:20:15.8411524Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-11-23T01:20:15.8411968Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6
2022-11-23T01:20:15.8412277Z UCC_HOME=/usr
2022-11-23T01:20:15.8412639Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:20:15.8412984Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2022-11-23T01:20:15.8413492Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1
2022-11-23T01:20:15.8414274Z INSTALLED_DB=yes
2022-11-23T01:20:15.8414629Z HOSTNAME=846842cde4cc
2022-11-23T01:20:15.8415116Z GITHUB_REF_NAME=master
2022-11-23T01:20:15.8415690Z GITHUB_API_URL=https://api.github.com
2022-11-23T01:20:15.8416249Z OPENSSL_DIR=/opt/openssl
2022-11-23T01:20:15.8416813Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2022-11-23T01:20:15.8418535Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_6a422a39-15b1-4bba-b8c2-23be0deb147e
2022-11-23T01:20:15.8419128Z CUDA_PATH=/usr/local/cuda
2022-11-23T01:20:15.8419704Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-11-23T01:20:15.8420512Z GITHUB_RUN_ATTEMPT=1
2022-11-23T01:20:15.8420971Z TEST_CONFIG=distributed
2022-11-23T01:20:15.8421528Z NV_LIBNPP_VERSION=11.6.3.124-1
2022-11-23T01:20:15.8422364Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1
2022-11-23T01:20:15.8422758Z GITHUB_REPOSITORY_OWNER=pytorch
2022-11-23T01:20:15.8423036Z GITHUB_ACTIONS=true
2022-11-23T01:20:15.8423512Z NVIDIA_VISIBLE_DEVICES=all
2022-11-23T01:20:15.8423971Z NV_NVPROF_VERSION=11.6.124-1
2022-11-23T01:20:15.8424271Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1
2022-11-23T01:20:15.8424528Z CI=true
2022-11-23T01:20:15.8424776Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-11-23T01:20:15.8425159Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1
2022-11-23T01:20:15.8425461Z BRANCH=master
2022-11-23T01:20:15.8425698Z GITHUB_HEAD_REF=
2022-11-23T01:20:15.8425986Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2022-11-23T01:20:15.8426309Z GITHUB_ACTOR=pytorchmergebot
2022-11-23T01:20:15.8426630Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2022-11-23T01:20:15.8427070Z GITHUB_ACTION_REF=
2022-11-23T01:20:15.8427346Z NCCL_VERSION=2.12.10-1
2022-11-23T01:20:15.8427598Z GITHUB_ACTION=__self
2022-11-23T01:20:15.8427824Z VALGRIND=ON
2022-11-23T01:20:15.8428075Z GITHUB_REF_PROTECTED=true
2022-11-23T01:20:15.8428527Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:20:15.8428894Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2022-11-23T01:20:15.8429525Z ***
2022-11-23T01:20:15.8429763Z INSTALLED_VISION=yes
2022-11-23T01:20:15.8430009Z NVARCH=x86_64
2022-11-23T01:20:15.8430300Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1
2022-11-23T01:20:15.8430577Z HOME=/var/lib/jenkins
2022-11-23T01:20:15.8431107Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_6a422a39-15b1-4bba-b8c2-23be0deb147e
2022-11-23T01:20:15.8431498Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-11-23T01:20:15.8431777Z GITHUB_ACTION_REPOSITORY=
2022-11-23T01:20:15.8432041Z GITHUB_REF_TYPE=branch
2022-11-23T01:20:15.8432332Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1
2022-11-23T01:20:15.8432621Z GITHUB_RETENTION_DAYS=90
2022-11-23T01:20:15.8433001Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-11-23T01:20:15.8433397Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6
2022-11-23T01:20:15.8433947Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_6a422a39-15b1-4bba-b8c2-23be0deb147e
2022-11-23T01:20:15.8434348Z DEBIAN_FRONTEND=noninteractive
2022-11-23T01:20:15.8434694Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2022-11-23T01:20:15.8434968Z GITHUB_REF=refs/heads/master
2022-11-23T01:20:15.8435259Z NV_CUDA_LIB_VERSION=11.6.2-1
2022-11-23T01:20:15.8435557Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:20:15.8435831Z INSTALLED_PROTOBUF=yes
2022-11-23T01:20:15.8436094Z GITHUB_RUN_ID=3528293562
2022-11-23T01:20:15.8436436Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1
2022-11-23T01:20:15.8436723Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2022-11-23T01:20:15.8437020Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2022-11-23T01:20:15.8437337Z NV_NVTX_VERSION=11.6.124-1
2022-11-23T01:20:15.8437620Z GITHUB_SERVER_URL=https://github.com
2022-11-23T01:20:15.8437896Z MAX_JOBS=30
2022-11-23T01:20:15.8438176Z NV_LIBCUBLAS_VERSION=11.9.2.110-1
2022-11-23T01:20:15.8438533Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1
2022-11-23T01:20:15.8439126Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-11-23T01:20:15.8439484Z UCX_HOME=/usr
2022-11-23T01:20:15.8439721Z PYTORCH_RETRY_TEST_CASES=1
2022-11-23T01:20:15.8440050Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-11-23T01:20:15.8440395Z BASE_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:20:15.8440730Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1
2022-11-23T01:20:15.8440979Z PR_BODY=
2022-11-23T01:20:15.8441210Z GITHUB_BASE_REF=
2022-11-23T01:20:15.8441445Z TERM=xterm
2022-11-23T01:20:15.8441651Z XLA_CUDA=
2022-11-23T01:20:15.8441921Z NV_NVML_DEV_VERSION=11.6.55-1
2022-11-23T01:20:15.8442193Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-11-23T01:20:15.8442440Z CUDA_VERSION=11.6.2
2022-11-23T01:20:15.8442778Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6
2022-11-23T01:20:15.8443077Z OPENSSL_ROOT_DIR=/opt/openssl
2022-11-23T01:20:15.8443599Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_6a422a39-15b1-4bba-b8c2-23be0deb147e
2022-11-23T01:20:15.8443982Z GITHUB_JOB=test
2022-11-23T01:20:15.8444238Z SCCACHE_S3_KEY_PREFIX=pull
2022-11-23T01:20:15.8444481Z COMMIT_MESSAGES=
2022-11-23T01:20:15.8444766Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2022-11-23T01:20:15.8445056Z NUM_TEST_SHARDS=3
2022-11-23T01:20:15.8445276Z PR_NUMBER=
2022-11-23T01:20:15.8445793Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_6a422a39-15b1-4bba-b8c2-23be0deb147e
2022-11-23T01:20:15.8446169Z SHLVL=1
2022-11-23T01:20:15.8446491Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6
2022-11-23T01:20:15.8446819Z GITHUB_REPOSITORY=pytorch/pytorch
2022-11-23T01:20:15.8447610Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471
2022-11-23T01:20:15.8448477Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1
2022-11-23T01:20:15.8448778Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:20:15.8449073Z GITHUB_EVENT_NAME=push
2022-11-23T01:20:15.8449391Z NV_CUDA_CUDART_VERSION=11.6.55-1
2022-11-23T01:20:15.8449745Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-11-23T01:20:15.8450045Z GITHUB_RUN_NUMBER=67324
2022-11-23T01:20:15.8450318Z GITHUB_WORKFLOW=pull
2022-11-23T01:20:15.8450756Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:20:15.8451229Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1
2022-11-23T01:20:15.8451690Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:20:15.8452074Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2022-11-23T01:20:15.8452343Z _=/usr/bin/env
2022-11-23T01:20:15.8452647Z + echo 'Testing pytorch'
2022-11-23T01:20:15.8452911Z Testing pytorch
2022-11-23T01:20:15.8453179Z + export LANG=C.UTF-8
2022-11-23T01:20:15.8453457Z + LANG=C.UTF-8
2022-11-23T01:20:15.8453698Z + PR_NUMBER=
2022-11-23T01:20:15.8453947Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-11-23T01:20:15.8454257Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-11-23T01:20:15.8454675Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:20:15.8454987Z + [[ distributed == \s\l\o\w ]]
2022-11-23T01:20:15.8455415Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *slow-gradcheck* ]]
2022-11-23T01:20:15.8455877Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-11-23T01:20:15.8456228Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-11-23T01:20:15.8456563Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-11-23T01:20:15.8456869Z + [[ distributed == *crossref* ]]
2022-11-23T01:20:15.8457156Z + [[ distributed == *dynamo* ]]
2022-11-23T01:20:15.8457426Z + [[ distributed == *inductor* ]]
2022-11-23T01:20:15.8458258Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:20:15.8458864Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *-bazel-* ]]
2022-11-23T01:20:15.8459230Z + pip_install --user ninja==1.10.2
2022-11-23T01:20:15.8459618Z + pip install --progress-bar off --user ninja==1.10.2
2022-11-23T01:20:16.3984524Z Collecting ninja==1.10.2
2022-11-23T01:20:16.4271283Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-11-23T01:20:17.2956132Z Installing collected packages: ninja
2022-11-23T01:20:17.3056245Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-11-23T01:20:17.3056915Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T01:20:17.3144694Z Successfully installed ninja-1.10.2
2022-11-23T01:20:17.3828102Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:20:17.3828740Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:20:17.3830142Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *asan* ]]
2022-11-23T01:20:17.3830590Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-tsan* ]]
2022-11-23T01:20:17.3830947Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-11-23T01:20:17.3831268Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-11-23T01:20:17.3838169Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *tbb* ]]
2022-11-23T01:20:17.3853069Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-11-23T01:20:17.3853764Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-bazel-* ]]
2022-11-23T01:20:17.3854174Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-tsan* ]]
2022-11-23T01:20:17.3857386Z + cd test
2022-11-23T01:20:17.3857775Z + python -c 'import torch; print(torch.__config__.show())'
2022-11-23T01:20:19.0724875Z PyTorch built with:
2022-11-23T01:20:19.0725429Z   - GCC 7.5
2022-11-23T01:20:19.0725706Z   - C++ Version: 201402
2022-11-23T01:20:19.0726249Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:20:19.0726807Z   - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:20:19.0727202Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:20:19.0727557Z   - LAPACK is enabled (usually provided by MKL)
2022-11-23T01:20:19.0727884Z   - NNPACK is enabled
2022-11-23T01:20:19.0728194Z   - CPU capability usage: AVX2
2022-11-23T01:20:19.0728481Z   - CUDA Runtime 11.6
2022-11-23T01:20:19.0728881Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-11-23T01:20:19.0729286Z   - CuDNN 8.3.2  (built against CUDA 11.5)
2022-11-23T01:20:19.0729569Z   - Magma 2.6.1
2022-11-23T01:20:19.0732847Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-11-23T01:20:19.0735010Z 
2022-11-23T01:20:19.3133045Z + cd test
2022-11-23T01:20:19.3133616Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-11-23T01:20:20.9286958Z ATen/Parallel:
2022-11-23T01:20:20.9287452Z 	at::get_num_threads() : 16
2022-11-23T01:20:20.9287977Z 	at::get_num_interop_threads() : 16
2022-11-23T01:20:20.9288261Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:20:20.9288540Z 	omp_get_max_threads() : 16
2022-11-23T01:20:20.9289172Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:20:20.9289556Z 	mkl_get_max_threads() : 16
2022-11-23T01:20:20.9289982Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:20:20.9290331Z std::thread::hardware_concurrency() : 32
2022-11-23T01:20:20.9290628Z Environment variables:
2022-11-23T01:20:20.9290890Z 	OMP_NUM_THREADS : [not set]
2022-11-23T01:20:20.9291156Z 	MKL_NUM_THREADS : [not set]
2022-11-23T01:20:20.9291425Z ATen parallel backend: OpenMP
2022-11-23T01:20:20.9291584Z 
2022-11-23T01:20:21.1550507Z + [[ distributed == *backward* ]]
2022-11-23T01:20:21.1550948Z + [[ distributed == *xla* ]]
2022-11-23T01:20:21.1551342Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-11-23T01:20:21.1551866Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-11-23T01:20:21.1552204Z + [[ distributed == distributed ]]
2022-11-23T01:20:21.1552471Z + install_filelock
2022-11-23T01:20:21.1552705Z + pip_install filelock
2022-11-23T01:20:21.1553064Z + pip install --progress-bar off filelock
2022-11-23T01:20:21.6889411Z Collecting filelock
2022-11-23T01:20:21.7079456Z   Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
2022-11-23T01:20:22.6079080Z Installing collected packages: filelock
2022-11-23T01:20:22.6474353Z Successfully installed filelock-3.8.0
2022-11-23T01:20:22.7142333Z + install_triton
2022-11-23T01:20:22.7142610Z + local commit
2022-11-23T01:20:22.7142867Z + [[ distributed == *rocm* ]]
2022-11-23T01:20:22.7146388Z ++ get_pinned_commit triton
2022-11-23T01:20:22.7147225Z ++ cat .github/ci_commit_pins/triton.txt
2022-11-23T01:20:22.7161913Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:20:22.7162828Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:20:22.7163538Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:20:23.1842836Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:20:23.1848704Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-5npkqwuz
2022-11-23T01:20:23.1869248Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-5npkqwuz
2022-11-23T01:20:23.9972132Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2022-11-23T01:20:23.9993176Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:20:24.4288853Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:20:24.6300658Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:20:24.6303383Z   Running command git submodule update --init --recursive -q
2022-11-23T01:20:25.2557822Z   Preparing metadata (setup.py) ... [?25l- done
2022-11-23T01:20:25.4673998Z [?25hCollecting cmake
2022-11-23T01:20:25.4882771Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2022-11-23T01:20:25.9658629Z Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (3.8.0)
2022-11-23T01:20:25.9661560Z Requirement already satisfied: torch in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385)
2022-11-23T01:20:25.9917963Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (4.4.0)
2022-11-23T01:20:25.9922512Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (2.6.3)
2022-11-23T01:20:25.9927026Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (1.11.1)
2022-11-23T01:20:26.0140370Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2022-11-23T01:20:26.0211623Z Building wheels for collected packages: triton
2022-11-23T01:21:18.6702147Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / done
2022-11-23T01:21:18.7183558Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp310-cp310-linux_x86_64.whl size=15377935 sha256=fb1bd5daa31132225e4f279a147b743fa485eb980828ffc3b443282a9d92a971
2022-11-23T01:21:18.7184952Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3f/1d/23/1c2bc47d618a44f9c949aea4b7e355e737a1f1ed208f009295
2022-11-23T01:21:18.7204832Z Successfully built triton
2022-11-23T01:21:19.5770636Z Installing collected packages: cmake, triton
2022-11-23T01:21:23.7198413Z Successfully installed cmake-3.25.0 triton-2.0.0
2022-11-23T01:21:23.8238952Z + pip_install --user jinja2
2022-11-23T01:21:23.8239366Z + pip install --progress-bar off --user jinja2
2022-11-23T01:21:24.7348972Z Collecting jinja2
2022-11-23T01:21:24.7571150Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2022-11-23T01:21:24.7911664Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2) (2.1.1)
2022-11-23T01:21:25.6877001Z Installing collected packages: jinja2
2022-11-23T01:21:25.7952597Z Successfully installed jinja2-3.1.2
2022-11-23T01:21:25.8638348Z + test_distributed
2022-11-23T01:21:25.8638803Z + echo 'Testing distributed python tests'
2022-11-23T01:21:25.8639118Z Testing distributed python tests
2022-11-23T01:21:25.8639556Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose
2022-11-23T01:21:28.0662429Z Ignoring disabled issues:  []
2022-11-23T01:21:28.1056132Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T01:21:28.1056702Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T01:21:28.1061459Z Found test time stats from artifacts
2022-11-23T01:21:28.1077466Z Selected tests:
2022-11-23T01:21:28.1077791Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:21:28.1078108Z  distributed/test_distributed_spawn
2022-11-23T01:21:28.1081121Z  distributed/rpc/test_tensorpipe_agent
2022-11-23T01:21:28.1081495Z  distributed/pipeline/sync/test_transparency
2022-11-23T01:21:28.1081844Z  distributed/pipeline/sync/test_pipe
2022-11-23T01:21:28.1082164Z  distributed/pipeline/sync/test_inplace
2022-11-23T01:21:28.1082456Z  distributed/pipeline/sync/test_copy
2022-11-23T01:21:28.1082766Z  distributed/pipeline/sync/test_balance
2022-11-23T01:21:28.1083093Z  distributed/pipeline/sync/skip/test_stash_pop
2022-11-23T01:21:28.1083444Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-11-23T01:21:28.1083956Z  distributed/optim/test_apply_optimizer_in_backward
2022-11-23T01:21:28.1084582Z  distributed/elastic/events/lib_test
2022-11-23T01:21:28.1085056Z  distributed/_shard/test_sharder
2022-11-23T01:21:28.1085598Z  distributed/test_launcher
2022-11-23T01:21:28.1086263Z  distributed/checkpoint/test_planner
2022-11-23T01:21:28.1086690Z  distributed/_composable/test_contract
2022-11-23T01:21:28.1086984Z  distributed/test_multi_threaded_pg
2022-11-23T01:21:28.1087285Z  distributed/elastic/utils/util_test
2022-11-23T01:21:28.1087777Z  distributed/rpc/test_share_memory
2022-11-23T01:21:28.1088270Z  distributed/fsdp/test_fsdp_traversal
2022-11-23T01:21:28.1088936Z  distributed/fsdp/test_fsdp_fx
2022-11-23T01:21:28.1089554Z  distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T01:21:28.1089895Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T01:21:28.1090259Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T01:21:28.1090597Z  distributed/_composable/test_replicate
2022-11-23T01:21:28.1090913Z  distributed/_shard/sharded_tensor/ops/test_init
2022-11-23T01:21:28.1091256Z  distributed/fsdp/test_distributed_checkpoint
2022-11-23T01:21:28.1092707Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T01:21:28.1093088Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-11-23T01:21:28.1093437Z  distributed/checkpoint/test_file_system_checkpoint_cpu
2022-11-23T01:21:28.1093887Z  distributed/fsdp/test_fsdp_apply
2022-11-23T01:21:28.1094470Z  distributed/_tensor/test_tp_sharding_ops
2022-11-23T01:21:28.1094809Z  distributed/_tensor/parallel/test_tp_style
2022-11-23T01:21:28.1095157Z  distributed/checkpoint/test_file_system_checkpoint
2022-11-23T01:21:28.1095488Z  distributed/checkpoint/test_checkpoint
2022-11-23T01:21:28.1095806Z  distributed/_tensor/test_redistribute
2022-11-23T01:21:28.1096091Z  distributed/_tensor/test_common_rules
2022-11-23T01:21:28.1096382Z  distributed/test_c10d_common
2022-11-23T01:21:28.1096701Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-11-23T01:21:28.1096994Z  distributed/fsdp/test_fsdp_comm
2022-11-23T01:21:28.1097313Z  distributed/fsdp/test_fsdp_freezing_weights
2022-11-23T01:21:28.1097625Z  distributed/test_store
2022-11-23T01:21:28.1098349Z  distributed/_tensor/test_tensor_ops
2022-11-23T01:21:28.1098976Z  distributed/fsdp/test_wrap
2022-11-23T01:21:28.1099401Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T01:21:28.1099731Z  distributed/fsdp/test_fsdp_summon_full_params
2022-11-23T01:21:28.1100051Z  distributed/fsdp/test_fsdp_state_dict
2022-11-23T01:21:28.1100345Z  distributed/test_c10d_nccl
2022-11-23T01:21:28.1100639Z Prioritized test from test file changes.
2022-11-23T01:21:28.1100952Z reordering tests for PR:
2022-11-23T01:21:28.1101216Z prioritized: []
2022-11-23T01:21:28.1105753Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/rpc/test_tensorpipe_agent', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/optim/test_apply_optimizer_in_backward', 'distributed/elastic/events/lib_test', 'distributed/_shard/test_sharder', 'distributed/test_launcher', 'distributed/checkpoint/test_planner', 'distributed/_composable/test_contract', 'distributed/test_multi_threaded_pg', 'distributed/elastic/utils/util_test', 'distributed/rpc/test_share_memory', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_fsdp_fx', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_composable/test_replicate', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/checkpoint/test_file_system_checkpoint_cpu', 'distributed/fsdp/test_fsdp_apply', 'distributed/_tensor/test_tp_sharding_ops', 'distributed/_tensor/parallel/test_tp_style', 'distributed/checkpoint/test_file_system_checkpoint', 'distributed/checkpoint/test_checkpoint', 'distributed/_tensor/test_redistribute', 'distributed/_tensor/test_common_rules', 'distributed/test_c10d_common', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/test_store', 'distributed/_tensor/test_tensor_ops', 'distributed/fsdp/test_wrap', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/test_c10d_nccl']
2022-11-23T01:21:28.1108651Z 
2022-11-23T01:21:28.1109206Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T01:21:28.1326530Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T01:21:28.1526172Z parallel (file granularity) tests:
2022-11-23T01:21:28.1526436Z  
2022-11-23T01:21:28.1526683Z serial (file granularity) tests:
2022-11-23T01:21:28.1527023Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:21:28.1527388Z  distributed/test_distributed_spawn
2022-11-23T01:21:28.1527687Z  distributed/rpc/test_tensorpipe_agent
2022-11-23T01:21:28.1528019Z  distributed/pipeline/sync/test_transparency
2022-11-23T01:21:28.1528343Z  distributed/pipeline/sync/test_pipe
2022-11-23T01:21:28.1528642Z  distributed/pipeline/sync/test_inplace
2022-11-23T01:21:28.1528956Z  distributed/pipeline/sync/test_copy
2022-11-23T01:21:28.1529266Z  distributed/pipeline/sync/test_balance
2022-11-23T01:21:28.1529577Z  distributed/pipeline/sync/skip/test_stash_pop
2022-11-23T01:21:28.1529938Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-11-23T01:21:28.1530302Z  distributed/optim/test_apply_optimizer_in_backward
2022-11-23T01:21:28.1530766Z  distributed/elastic/events/lib_test
2022-11-23T01:21:28.1531064Z  distributed/_shard/test_sharder
2022-11-23T01:21:28.1531342Z  distributed/test_launcher
2022-11-23T01:21:28.1531615Z  distributed/checkpoint/test_planner
2022-11-23T01:21:28.1531928Z  distributed/_composable/test_contract
2022-11-23T01:21:28.1532236Z  distributed/test_multi_threaded_pg
2022-11-23T01:21:28.1532519Z  distributed/elastic/utils/util_test
2022-11-23T01:21:28.1532841Z  distributed/rpc/test_share_memory
2022-11-23T01:21:28.1533142Z  distributed/fsdp/test_fsdp_traversal
2022-11-23T01:21:28.1533440Z  distributed/fsdp/test_fsdp_fx
2022-11-23T01:21:28.1533733Z  distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T01:21:28.1534082Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T01:21:28.1534447Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T01:21:28.1534767Z  distributed/_composable/test_replicate
2022-11-23T01:21:28.1535098Z  distributed/_shard/sharded_tensor/ops/test_init
2022-11-23T01:21:28.1535446Z  distributed/fsdp/test_distributed_checkpoint
2022-11-23T01:21:28.1535778Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T01:21:28.1536137Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-11-23T01:21:28.1536505Z  distributed/checkpoint/test_file_system_checkpoint_cpu
2022-11-23T01:21:28.1536845Z  distributed/fsdp/test_fsdp_apply
2022-11-23T01:21:28.1537199Z  distributed/_tensor/test_tp_sharding_ops
2022-11-23T01:21:28.1537536Z  distributed/_tensor/parallel/test_tp_style
2022-11-23T01:21:28.1537875Z  distributed/checkpoint/test_file_system_checkpoint
2022-11-23T01:21:28.1538454Z  distributed/checkpoint/test_checkpoint
2022-11-23T01:21:28.1538799Z  distributed/_tensor/test_redistribute
2022-11-23T01:21:28.1539109Z  distributed/_tensor/test_common_rules
2022-11-23T01:21:28.1539409Z  distributed/test_c10d_common
2022-11-23T01:21:28.1539722Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-11-23T01:21:28.1540035Z  distributed/fsdp/test_fsdp_comm
2022-11-23T01:21:28.1540338Z  distributed/fsdp/test_fsdp_freezing_weights
2022-11-23T01:21:28.1540636Z  distributed/test_store
2022-11-23T01:21:28.1540919Z  distributed/_tensor/test_tensor_ops
2022-11-23T01:21:28.1541191Z  distributed/fsdp/test_wrap
2022-11-23T01:21:28.1541509Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T01:21:28.1541950Z  distributed/fsdp/test_fsdp_summon_full_params
2022-11-23T01:21:28.1542267Z  distributed/fsdp/test_fsdp_state_dict
2022-11-23T01:21:28.1542558Z  distributed/test_c10d_nccl
2022-11-23T01:21:30.4173093Z Ignoring disabled issues:  []
2022-11-23T01:21:30.4211068Z Ignoring disabled issues:  []
2022-11-23T01:21:30.8220574Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:21:30.821548]
2022-11-23T01:21:30.8228386Z /usr/bin/mpiexec
2022-11-23T01:21:30.8229146Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:21:30.8229728Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2022-11-23T01:21:30.8230163Z Shard 2: test should be run in 1
2022-11-23T01:21:30.8234128Z Running distributed tests for the nccl backend with env init_method in shard 2 of 3
2022-11-23T01:21:30.8239345Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:21:30.823647]
2022-11-23T01:22:12.4646879Z 
2022-11-23T01:22:12.4647429Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:22:12.4656841Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_eh44iml7)
2022-11-23T01:22:12.4658518Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:22:12.4659700Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:12.4660057Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:12.4660419Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:12.4660929Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:12.4661617Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:12.4662353Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:12.4663002Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:22:12.4663868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4664638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4665600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4666688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4667164Z 
2022-11-23T01:22:12.4667334Z Running tests...
2022-11-23T01:22:12.4668113Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4669284Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:12.4670486Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-11-23T01:22:12.4671141Z 
2022-11-23T01:22:12.4671630Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4671973Z Ran 1 test in 0.001s
2022-11-23T01:22:12.4672116Z 
2022-11-23T01:22:12.4672227Z OK (skipped=1)
2022-11-23T01:22:12.4672388Z 
2022-11-23T01:22:12.4672514Z Generating XML reports...
2022-11-23T01:22:12.4673175Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012134.xml
2022-11-23T01:22:12.4674084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4674561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4675141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4675607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4675835Z 
2022-11-23T01:22:12.4675926Z Running tests...
2022-11-23T01:22:12.4676324Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4676921Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:12.4677485Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-11-23T01:22:12.4677751Z 
2022-11-23T01:22:12.4678017Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4678347Z Ran 1 test in 0.001s
2022-11-23T01:22:12.4678511Z 
2022-11-23T01:22:12.4678622Z OK (skipped=1)
2022-11-23T01:22:12.4678774Z 
2022-11-23T01:22:12.4678879Z Generating XML reports...
2022-11-23T01:22:12.4679536Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012136.xml
2022-11-23T01:22:12.4680288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4680737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4681292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4681848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4682082Z 
2022-11-23T01:22:12.4682191Z Running tests...
2022-11-23T01:22:12.4682578Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4683185Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:12.4683726Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:12.4684194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 975
2022-11-23T01:22:12.4684620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 976
2022-11-23T01:22:12.4685214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4685665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4686222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4686684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4687267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4687711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4688260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4688717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4689149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:12.4689631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:12.4690101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:12.4690577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:12.4691294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:12.4691817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:22:12.4692466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:12.4692987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:22:12.4693628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:12.4694291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:12.4694684Z ok (6.672s)
2022-11-23T01:22:12.4694830Z 
2022-11-23T01:22:12.4695100Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4695426Z Ran 1 test in 6.672s
2022-11-23T01:22:12.4695572Z 
2022-11-23T01:22:12.4695671Z OK
2022-11-23T01:22:12.4695803Z 
2022-11-23T01:22:12.4695925Z Generating XML reports...
2022-11-23T01:22:12.4696580Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012138.xml
2022-11-23T01:22:12.4697316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4697767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4699019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4699614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4699826Z 
2022-11-23T01:22:12.4699938Z Running tests...
2022-11-23T01:22:12.4700351Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4700954Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:12.4701479Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:12.4701944Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1094
2022-11-23T01:22:12.4702389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1095
2022-11-23T01:22:12.4702992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4703422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4703992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4704465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4705026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4705513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4706084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4706548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4706967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:12.4707434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:12.4707913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:12.4708406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:12.4709126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:12.4709668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:22:12.4710317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:12.4710828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:22:12.4711469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:12.4712141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:12.4712538Z ok (6.660s)
2022-11-23T01:22:12.4712689Z 
2022-11-23T01:22:12.4712937Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4713264Z Ran 1 test in 6.660s
2022-11-23T01:22:12.4713422Z 
2022-11-23T01:22:12.4713514Z OK
2022-11-23T01:22:12.4713653Z 
2022-11-23T01:22:12.4713779Z Generating XML reports...
2022-11-23T01:22:12.4714420Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012147.xml
2022-11-23T01:22:12.4715176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4715621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4716175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4716642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4716954Z 
2022-11-23T01:22:12.4717065Z Running tests...
2022-11-23T01:22:12.4717468Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4718052Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:12.4718606Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:12.4719082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1213
2022-11-23T01:22:12.4719508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1214
2022-11-23T01:22:12.4720105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4720550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4721124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4721577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4722153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4722595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4723162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4723604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4724035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:12.4724501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:12.4724964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:12.4725456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:12.4726106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:12.4726699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:22:12.4727339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:12.4727866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:22:12.4728508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:12.4729185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:12.4729561Z ok (6.512s)
2022-11-23T01:22:12.4729715Z 
2022-11-23T01:22:12.4729978Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4730306Z Ran 1 test in 6.512s
2022-11-23T01:22:12.4730469Z 
2022-11-23T01:22:12.4730545Z OK
2022-11-23T01:22:12.4730680Z 
2022-11-23T01:22:12.4730809Z Generating XML reports...
2022-11-23T01:22:12.4731471Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012156.xml
2022-11-23T01:22:12.4732223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4732651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4733223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4733687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4733987Z 
2022-11-23T01:22:12.4734078Z Running tests...
2022-11-23T01:22:12.4734484Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4735084Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:12.4735640Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:12.4736098Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1332
2022-11-23T01:22:12.4736530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1333
2022-11-23T01:22:12.4737135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4737563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4738812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4739297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4739879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:12.4740305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:12.4740875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:12.4741339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:12.4741778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:12.4742225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:12.4742727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:12.4743220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:12.4743883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:12.4744491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:22:12.4745155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:12.4745675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:22:12.4746301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:12.4746981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:12.4747370Z ok (6.515s)
2022-11-23T01:22:12.4747518Z 
2022-11-23T01:22:12.4747792Z ----------------------------------------------------------------------
2022-11-23T01:22:12.4748107Z Ran 1 test in 6.515s
2022-11-23T01:22:12.4748268Z 
2022-11-23T01:22:12.4748362Z OK
2022-11-23T01:22:12.4748497Z 
2022-11-23T01:22:12.4748620Z Generating XML reports...
2022-11-23T01:22:12.4749265Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012205.xml
2022-11-23T01:22:12.4749671Z 
2022-11-23T01:22:12.4750024Z ##[endgroup]
2022-11-23T01:22:12.4750735Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_eh44iml7)
2022-11-23T01:22:12.4751171Z 
2022-11-23T01:22:12.4751386Z Running distributed tests for the nccl backend with file init_method in shard 2 of 3
2022-11-23T01:22:12.4752145Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:22:12.466407]
2022-11-23T01:22:54.1581968Z 
2022-11-23T01:22:54.1582531Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:22:54.1583613Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_xcs95p7u)
2022-11-23T01:22:54.1586644Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:22:54.1587622Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:54.1588002Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:54.1588367Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:54.1588722Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:54.1589095Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:54.1589482Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:54.1589826Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:22:54.1590507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1590947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1591532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1592009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1593112Z 
2022-11-23T01:22:54.1593460Z Running tests...
2022-11-23T01:22:54.1593979Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1594601Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:54.1595402Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-11-23T01:22:54.1595718Z 
2022-11-23T01:22:54.1595982Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1596291Z Ran 1 test in 0.001s
2022-11-23T01:22:54.1596452Z 
2022-11-23T01:22:54.1596563Z OK (skipped=1)
2022-11-23T01:22:54.1596722Z 
2022-11-23T01:22:54.1596862Z Generating XML reports...
2022-11-23T01:22:54.1598054Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012216.xml
2022-11-23T01:22:54.1598851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1599310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1599886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1600344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1600576Z 
2022-11-23T01:22:54.1600687Z Running tests...
2022-11-23T01:22:54.1601163Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1601767Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:54.1602326Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-11-23T01:22:54.1602614Z 
2022-11-23T01:22:54.1602875Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1603332Z Ran 1 test in 0.001s
2022-11-23T01:22:54.1603491Z 
2022-11-23T01:22:54.1603602Z OK (skipped=1)
2022-11-23T01:22:54.1603756Z 
2022-11-23T01:22:54.1603881Z Generating XML reports...
2022-11-23T01:22:54.1604523Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012218.xml
2022-11-23T01:22:54.1605329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1605775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1606343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1606790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1607206Z 
2022-11-23T01:22:54.1607419Z Running tests...
2022-11-23T01:22:54.1607851Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1608604Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:54.1609147Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:54.1609625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1550
2022-11-23T01:22:54.1610101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1551
2022-11-23T01:22:54.1610953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1611405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1611980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1612445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1612998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1613445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1614102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1614702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1615285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:54.1615757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:54.1616237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:54.1616706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:54.1617361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:54.1618427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:22:54.1619171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:54.1619679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:22:54.1620315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:54.1620987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:54.1621362Z ok (6.607s)
2022-11-23T01:22:54.1621508Z 
2022-11-23T01:22:54.1621772Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1622097Z Ran 1 test in 6.607s
2022-11-23T01:22:54.1622375Z 
2022-11-23T01:22:54.1622469Z OK
2022-11-23T01:22:54.1622585Z 
2022-11-23T01:22:54.1622711Z Generating XML reports...
2022-11-23T01:22:54.1623367Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012220.xml
2022-11-23T01:22:54.1624121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1624557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1625126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1625593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1625818Z 
2022-11-23T01:22:54.1625927Z Running tests...
2022-11-23T01:22:54.1626309Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1626901Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:54.1627445Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:54.1627908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1669
2022-11-23T01:22:54.1628331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1670
2022-11-23T01:22:54.1628931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1629372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1629922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1630381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1630952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1631396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1631942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1632469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1632907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:54.1633356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:54.1633833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:54.1634320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:54.1634972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:54.1635483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:22:54.1636123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:54.1636645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:22:54.1637283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:54.1637940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:54.1638333Z ok (6.509s)
2022-11-23T01:22:54.1638480Z 
2022-11-23T01:22:54.1638745Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1639051Z Ran 1 test in 6.509s
2022-11-23T01:22:54.1639213Z 
2022-11-23T01:22:54.1639301Z OK
2022-11-23T01:22:54.1639498Z 
2022-11-23T01:22:54.1639623Z Generating XML reports...
2022-11-23T01:22:54.1640284Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012229.xml
2022-11-23T01:22:54.1641020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1641462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1642025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1642473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1642701Z 
2022-11-23T01:22:54.1642809Z Running tests...
2022-11-23T01:22:54.1643214Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1643806Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:54.1644342Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:54.1644815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1788
2022-11-23T01:22:54.1645258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1789
2022-11-23T01:22:54.1645860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1646287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1646852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1647315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1647866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1648305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1648873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1649335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1649798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:54.1650285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:54.1650763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:54.1651219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:54.1651871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:54.1652398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:22:54.1653035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:54.1653549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:22:54.1654183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:54.1654854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:54.1655240Z ok (6.646s)
2022-11-23T01:22:54.1655369Z 
2022-11-23T01:22:54.1655635Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1655957Z Ran 1 test in 6.646s
2022-11-23T01:22:54.1656116Z 
2022-11-23T01:22:54.1656209Z OK
2022-11-23T01:22:54.1656345Z 
2022-11-23T01:22:54.1656452Z Generating XML reports...
2022-11-23T01:22:54.1657174Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012237.xml
2022-11-23T01:22:54.1658957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1659437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1659998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1660459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1660685Z 
2022-11-23T01:22:54.1660795Z Running tests...
2022-11-23T01:22:54.1661179Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1661777Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:54.1662327Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:54.1662806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1907
2022-11-23T01:22:54.1663232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1908
2022-11-23T01:22:54.1663829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1664272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1664846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1665293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1665862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:54.1666300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:54.1666852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:54.1667310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:54.1667825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:54.1668306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:54.1668766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:54.1669252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:54.1669905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:54.1670417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:22:54.1671059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:54.1671583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:22:54.1672227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:54.1672880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:22:54.1673267Z ok (6.649s)
2022-11-23T01:22:54.1673415Z 
2022-11-23T01:22:54.1673679Z ----------------------------------------------------------------------
2022-11-23T01:22:54.1674002Z Ran 1 test in 6.650s
2022-11-23T01:22:54.1674143Z 
2022-11-23T01:22:54.1674238Z OK
2022-11-23T01:22:54.1674370Z 
2022-11-23T01:22:54.1674496Z Generating XML reports...
2022-11-23T01:22:54.1675155Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012246.xml
2022-11-23T01:22:54.1675635Z 
2022-11-23T01:22:54.1676029Z ##[endgroup]
2022-11-23T01:22:54.1676741Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_xcs95p7u)
2022-11-23T01:22:54.1677173Z 
2022-11-23T01:22:54.1677302Z Shard 2: gloo should be run in 1
2022-11-23T01:22:54.1677582Z Shard 2: ucc should be run in 1
2022-11-23T01:22:54.1677982Z Running distributed/test_distributed_spawn ... [2022-11-23 01:22:54.159582]
2022-11-23T01:22:54.1678310Z /usr/bin/mpiexec
2022-11-23T01:22:54.1678672Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:22:54.1679202Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 3}
2022-11-23T01:22:54.1679588Z Shard 2: test should be run in 1
2022-11-23T01:22:54.1679958Z Running distributed tests for the nccl backend with env init_method in shard 2 of 3
2022-11-23T01:22:54.1680696Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:22:54.161343]
2022-11-23T01:48:47.5245130Z 
2022-11-23T01:48:47.5247630Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:48:47.5248573Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_u1jzs9hx)
2022-11-23T01:48:47.5252149Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:48:47.5304771Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T01:48:47.5367299Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5410279Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5410732Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5411155Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5411596Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5412068Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5412565Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5413061Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5413557Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5414109Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5414671Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5415197Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5415714Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5416236Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5416862Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5417355Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5417825Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5418992Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5419428Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5419851Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5420331Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5420807Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5421205Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5421603Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5422037Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5422460Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5422860Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5423282Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5423691Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5424054Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5424444Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5424848Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5425340Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5425744Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5426170Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5426590Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5426977Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5427397Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5427814Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5428199Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5428611Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5429055Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5429503Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5429938Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5430368Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5430797Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5431209Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5431636Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5432051Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5432491Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5432908Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5433316Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5433726Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5434131Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5434550Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5435033Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5435438Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5435845Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5436244Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5436630Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5437006Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5437403Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5437779Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5438133Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5438523Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5438927Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5439313Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5439704Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5440083Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5440461Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5440832Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5441223Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5441614Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5441999Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5442380Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5442812Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5443170Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5443553Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5443953Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5444348Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5444719Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5445103Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5445503Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5445911Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5446342Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5446781Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5447234Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5447665Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5448111Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5448548Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5448961Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5449389Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5449828Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5450270Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5450706Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5451167Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5451616Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5452094Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5452526Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5452914Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5453289Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5453633Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5453994Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5454374Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5454746Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5455133Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5455510Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5455883Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5456286Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5456681Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5457057Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5457458Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5458342Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5458931Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5459328Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5459736Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5460232Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5460642Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5461054Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5461461Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5461848Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5462197Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5462584Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5462968Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5463336Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5463724Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5464172Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5464665Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5465114Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5465528Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5465938Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5466383Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5466827Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5467289Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5467707Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5468129Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5468566Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5468973Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5469322Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5469706Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5470187Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5470591Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5471021Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5471454Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5471848Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5472256Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5472714Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5473210Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5473749Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5474360Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5474969Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5475569Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5476148Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5476813Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5477414Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5478009Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5478533Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5479021Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5479459Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5479842Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5480218Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5480626Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5481020Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5481420Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5481858Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5482315Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5482778Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5483169Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5483551Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5483960Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5484376Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5484805Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5485216Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5485605Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5486077Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5486511Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5486926Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5487313Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5487716Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5488131Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5488511Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5488923Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5489379Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5489808Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5490185Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5490562Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5490956Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5491352Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5491759Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5492136Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5492477Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5492844Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5493218Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5493653Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5494003Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5494389Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5494773Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5495112Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5495464Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5495841Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5496214Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5496600Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5496967Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5497292Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5497669Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5498705Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5499121Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5499593Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5500045Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5500454Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5500850Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5501279Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5501702Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5502119Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5502509Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5502922Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5503327Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5503717Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5504191Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5504598Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5505043Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5505527Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5505992Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5506435Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5506872Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5507321Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5507745Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5508148Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5508571Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5509006Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5509450Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5509905Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5510412Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5510888Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5511367Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5511769Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5512166Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5512570Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5512936Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5513297Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5513669Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5514033Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5514400Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5514756Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5515102Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5515483Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5515880Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5516277Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5516633Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5517001Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5517379Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5517744Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5518107Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5518512Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5518868Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5519241Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5519625Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5520019Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5520378Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5520756Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5521207Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5521573Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5521987Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5522429Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5522832Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5523217Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5523612Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5524029Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5524428Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5524820Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5525225Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5525642Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5526049Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5526449Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5526849Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5527227Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5527605Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5528018Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5528439Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5528936Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5529676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5530109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5530679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5531143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5531372Z 
2022-11-23T01:48:47.5531483Z Running tests...
2022-11-23T01:48:47.5531872Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5532398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5532976Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5533535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2060
2022-11-23T01:48:47.5533961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2061
2022-11-23T01:48:47.5534559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5535001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5535553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5536019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5536589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5537027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5537580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5538757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5539324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5539808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5540474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5541161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5541681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5542129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5542651Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:47.5543480Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:47.5544141Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:47.5544944Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:47.5545596Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:47.5546405Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:47.5547132Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:47.5547932Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:47.5548587Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:47.5549388Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:47.5550035Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:47.5550831Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:47.5551486Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:47.5552295Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:47.5552943Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:47.5553753Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:47.5554218Z ok (6.854s)
2022-11-23T01:48:47.5554368Z 
2022-11-23T01:48:47.5554637Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5554966Z Ran 1 test in 6.854s
2022-11-23T01:48:47.5555125Z 
2022-11-23T01:48:47.5555205Z OK
2022-11-23T01:48:47.5555339Z 
2022-11-23T01:48:47.5555463Z Generating XML reports...
2022-11-23T01:48:47.5556071Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012258.xml
2022-11-23T01:48:47.5556832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5557272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5557847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5558314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5558543Z 
2022-11-23T01:48:47.5558635Z Running tests...
2022-11-23T01:48:47.5559038Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5559565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5560095Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-11-23T01:48:47.5560379Z 
2022-11-23T01:48:47.5560643Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5560970Z Ran 1 test in 0.004s
2022-11-23T01:48:47.5561129Z 
2022-11-23T01:48:47.5561240Z OK (skipped=1)
2022-11-23T01:48:47.5561392Z 
2022-11-23T01:48:47.5561497Z Generating XML reports...
2022-11-23T01:48:47.5562102Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012307.xml
2022-11-23T01:48:47.5562815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5563260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5563817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5564348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5564573Z 
2022-11-23T01:48:47.5564683Z Running tests...
2022-11-23T01:48:47.5565069Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5565594Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5566097Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5566575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2208
2022-11-23T01:48:47.5567001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2209
2022-11-23T01:48:47.5567595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5568039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5568628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5569076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5569655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5570094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5570640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5571103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5571551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5572045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5572682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5573375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5573942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5574416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5574740Z ok (4.246s)
2022-11-23T01:48:47.5574888Z 
2022-11-23T01:48:47.5575155Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5575481Z Ran 1 test in 4.246s
2022-11-23T01:48:47.5575640Z 
2022-11-23T01:48:47.5575714Z OK
2022-11-23T01:48:47.5575845Z 
2022-11-23T01:48:47.5575969Z Generating XML reports...
2022-11-23T01:48:47.5576571Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012310.xml
2022-11-23T01:48:47.5577271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5577722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5578829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5579296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5579524Z 
2022-11-23T01:48:47.5579615Z Running tests...
2022-11-23T01:48:47.5580020Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5580546Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5581072Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5582107Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.643s)
2022-11-23T01:48:47.5582732Z 
2022-11-23T01:48:47.5583007Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5583332Z Ran 1 test in 1.644s
2022-11-23T01:48:47.5583491Z 
2022-11-23T01:48:47.5583600Z OK (skipped=1)
2022-11-23T01:48:47.5583755Z 
2022-11-23T01:48:47.5583860Z Generating XML reports...
2022-11-23T01:48:47.5584462Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012317.xml
2022-11-23T01:48:47.5585167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5585617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5586169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5586639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5586864Z 
2022-11-23T01:48:47.5586972Z Running tests...
2022-11-23T01:48:47.5587359Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5587883Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5588427Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T01:48:47.5588749Z 
2022-11-23T01:48:47.5589008Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5589313Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5589469Z 
2022-11-23T01:48:47.5589577Z OK (skipped=1)
2022-11-23T01:48:47.5589729Z 
2022-11-23T01:48:47.5589853Z Generating XML reports...
2022-11-23T01:48:47.5590441Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012321.xml
2022-11-23T01:48:47.5591151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5591670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5592253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5592701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5592928Z 
2022-11-23T01:48:47.5593037Z Running tests...
2022-11-23T01:48:47.5593436Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5593963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5594509Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T01:48:47.5594857Z 
2022-11-23T01:48:47.5595118Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5595441Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5595602Z 
2022-11-23T01:48:47.5595693Z OK (skipped=1)
2022-11-23T01:48:47.5595849Z 
2022-11-23T01:48:47.5595973Z Generating XML reports...
2022-11-23T01:48:47.5596573Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012323.xml
2022-11-23T01:48:47.5597279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5597706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5598274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5598741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5599024Z 
2022-11-23T01:48:47.5599115Z Running tests...
2022-11-23T01:48:47.5599564Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5600093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5600642Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5601147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2413
2022-11-23T01:48:47.5601593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2414
2022-11-23T01:48:47.5602194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5602641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5603179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5603628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5604197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5604648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5605230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5605691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5606137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5606616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5607268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5607957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5608473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5608979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5609477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v75mfpa
2022-11-23T01:48:47.5610016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v75mfpa/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5610524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0pv3i1mb
2022-11-23T01:48:47.5611049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0pv3i1mb/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5611956Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5612508Z   warnings.warn(
2022-11-23T01:48:47.5613249Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5613778Z   warnings.warn(
2022-11-23T01:48:47.5614157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5614640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5615096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5615572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5616040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5616551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5617023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5617496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5618442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5618928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5619389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5619853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5620177Z ok (8.043s)
2022-11-23T01:48:47.5620326Z 
2022-11-23T01:48:47.5620615Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5620949Z Ran 1 test in 8.043s
2022-11-23T01:48:47.5621109Z 
2022-11-23T01:48:47.5621203Z OK
2022-11-23T01:48:47.5621319Z 
2022-11-23T01:48:47.5621443Z Generating XML reports...
2022-11-23T01:48:47.5622053Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012326.xml
2022-11-23T01:48:47.5622767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5623196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5623765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5624230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5624456Z 
2022-11-23T01:48:47.5624565Z Running tests...
2022-11-23T01:48:47.5624947Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5625478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5626033Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5626637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2532
2022-11-23T01:48:47.5627100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2533
2022-11-23T01:48:47.5627694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5628143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5628696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5629156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5629736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5630176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5630724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5631181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5631631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5632105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5632759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5633438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5634025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5634472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5634961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxmeqq2l
2022-11-23T01:48:47.5635497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxmeqq2l/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5636029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9gcyzn7
2022-11-23T01:48:47.5636539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9gcyzn7/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5637446Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5637996Z   warnings.warn(
2022-11-23T01:48:47.5638736Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5639246Z   warnings.warn(
2022-11-23T01:48:47.5639596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5640069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5640525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5640978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5641320Z ok (6.848s)
2022-11-23T01:48:47.5641464Z 
2022-11-23T01:48:47.5641736Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5642050Z Ran 1 test in 6.848s
2022-11-23T01:48:47.5642209Z 
2022-11-23T01:48:47.5642303Z OK
2022-11-23T01:48:47.5642438Z 
2022-11-23T01:48:47.5642562Z Generating XML reports...
2022-11-23T01:48:47.5643211Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012336.xml
2022-11-23T01:48:47.5643936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5644382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5644947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5645397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5645620Z 
2022-11-23T01:48:47.5645732Z Running tests...
2022-11-23T01:48:47.5646136Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5646651Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5647213Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5647756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2651
2022-11-23T01:48:47.5648201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2652
2022-11-23T01:48:47.5648786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5649232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5649803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5650249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5650816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5651312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5651878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5652318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5652764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5653255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5653901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5654565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5655090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5655559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5656041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtjwuk7z
2022-11-23T01:48:47.5656571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtjwuk7z/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5657100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwq8xxxy
2022-11-23T01:48:47.5657630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwq8xxxy/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5659122Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5659674Z   warnings.warn(
2022-11-23T01:48:47.5660433Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5661056Z   warnings.warn(
2022-11-23T01:48:47.5661416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5661876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5662349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5662826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5663275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5663739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5664213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5664660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5664999Z ok (7.145s)
2022-11-23T01:48:47.5665147Z 
2022-11-23T01:48:47.5665424Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5665752Z Ran 1 test in 7.146s
2022-11-23T01:48:47.5665893Z 
2022-11-23T01:48:47.5665986Z OK
2022-11-23T01:48:47.5666120Z 
2022-11-23T01:48:47.5666244Z Generating XML reports...
2022-11-23T01:48:47.5666847Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012345.xml
2022-11-23T01:48:47.5667539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5667985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5668681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5669144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5669351Z 
2022-11-23T01:48:47.5669460Z Running tests...
2022-11-23T01:48:47.5669865Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5670388Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5670954Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5671511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2770
2022-11-23T01:48:47.5671956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2771
2022-11-23T01:48:47.5672559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5672993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5673563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5674031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5674585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5675023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5675588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5676046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5676473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5676972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5677628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5678362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5678870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5679336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5679829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3o7lylx2
2022-11-23T01:48:47.5680345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3o7lylx2/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5680869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptis0_xv8
2022-11-23T01:48:47.5681400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptis0_xv8/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5682304Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5682832Z   warnings.warn(
2022-11-23T01:48:47.5683584Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5684121Z   warnings.warn(
2022-11-23T01:48:47.5684492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5684956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5685377Z ok (7.114s)
2022-11-23T01:48:47.5685523Z 
2022-11-23T01:48:47.5685795Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5686099Z Ran 1 test in 7.114s
2022-11-23T01:48:47.5686257Z 
2022-11-23T01:48:47.5686349Z OK
2022-11-23T01:48:47.5686483Z 
2022-11-23T01:48:47.5686613Z Generating XML reports...
2022-11-23T01:48:47.5687218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012355.xml
2022-11-23T01:48:47.5687912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5688357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5688927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5689379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5689607Z 
2022-11-23T01:48:47.5689720Z Running tests...
2022-11-23T01:48:47.5690121Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5690642Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5691209Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5691755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2889
2022-11-23T01:48:47.5692200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2890
2022-11-23T01:48:47.5692800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5693230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5693796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5694264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5694815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5695303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5695879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5696407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5696836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5697334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5698224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5698936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5699433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5699988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5700481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_yw95o9c
2022-11-23T01:48:47.5700992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_yw95o9c/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5701515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84bz8by9
2022-11-23T01:48:47.5702044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84bz8by9/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5702942Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5703557Z   warnings.warn(
2022-11-23T01:48:47.5704315Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5704856Z   warnings.warn(
2022-11-23T01:48:47.5705228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5705687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5706167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5706642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5706963Z ok (7.806s)
2022-11-23T01:48:47.5707114Z 
2022-11-23T01:48:47.5707387Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5707713Z Ran 1 test in 7.806s
2022-11-23T01:48:47.5707872Z 
2022-11-23T01:48:47.5707966Z OK
2022-11-23T01:48:47.5708080Z 
2022-11-23T01:48:47.5708204Z Generating XML reports...
2022-11-23T01:48:47.5708871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012405.xml
2022-11-23T01:48:47.5709585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5710012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5710584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5711049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5711275Z 
2022-11-23T01:48:47.5711382Z Running tests...
2022-11-23T01:48:47.5711772Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5712301Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5712928Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5713454Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3008
2022-11-23T01:48:47.5713898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3009
2022-11-23T01:48:47.5714500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5714947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5715497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5715964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5716535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5716974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5717522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5717978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5718420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5718895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5719547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5720234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5720810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5721262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5721747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ebq62o3
2022-11-23T01:48:47.5722277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ebq62o3/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5722800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3d04vxrv
2022-11-23T01:48:47.5723312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3d04vxrv/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5724213Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5724765Z   warnings.warn(
2022-11-23T01:48:47.5725525Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5726047Z   warnings.warn(
2022-11-23T01:48:47.5726416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5726894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5727350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5727822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5728165Z ok (7.554s)
2022-11-23T01:48:47.5728309Z 
2022-11-23T01:48:47.5728581Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5728892Z Ran 1 test in 7.555s
2022-11-23T01:48:47.5729049Z 
2022-11-23T01:48:47.5729144Z OK
2022-11-23T01:48:47.5729277Z 
2022-11-23T01:48:47.5729399Z Generating XML reports...
2022-11-23T01:48:47.5730041Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012415.xml
2022-11-23T01:48:47.5730765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5731212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5731780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5732228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5732454Z 
2022-11-23T01:48:47.5732564Z Running tests...
2022-11-23T01:48:47.5732965Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5733481Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5734063Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5734616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3127
2022-11-23T01:48:47.5735059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3128
2022-11-23T01:48:47.5735642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5736087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5736660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5737103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5737741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5738424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5739003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5739447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5739891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5740387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5741033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5741700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5742223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5742690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5743166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68e627sh
2022-11-23T01:48:47.5743699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68e627sh/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5744222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj72f98k3
2022-11-23T01:48:47.5744749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj72f98k3/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5745631Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5746176Z   warnings.warn(
2022-11-23T01:48:47.5747012Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.5747561Z   warnings.warn(
2022-11-23T01:48:47.5747919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5748399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5748877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5749348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5749673Z ok (6.813s)
2022-11-23T01:48:47.5749819Z 
2022-11-23T01:48:47.5750091Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5750419Z Ran 1 test in 6.813s
2022-11-23T01:48:47.5750578Z 
2022-11-23T01:48:47.5750654Z OK
2022-11-23T01:48:47.5750788Z 
2022-11-23T01:48:47.5750912Z Generating XML reports...
2022-11-23T01:48:47.5751522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012425.xml
2022-11-23T01:48:47.5752237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5752664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5753233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5753697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5753920Z 
2022-11-23T01:48:47.5754012Z Running tests...
2022-11-23T01:48:47.5754414Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5755018Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5755567Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5756612Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T01:48:47.5757126Z 
2022-11-23T01:48:47.5757388Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5757716Z Ran 1 test in 1.636s
2022-11-23T01:48:47.5757877Z 
2022-11-23T01:48:47.5757985Z OK (skipped=1)
2022-11-23T01:48:47.5758121Z 
2022-11-23T01:48:47.5758246Z Generating XML reports...
2022-11-23T01:48:47.5758845Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012434.xml
2022-11-23T01:48:47.5759559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5759989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5760560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5761020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5761246Z 
2022-11-23T01:48:47.5761352Z Running tests...
2022-11-23T01:48:47.5761733Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5762258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5762797Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5763320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3280
2022-11-23T01:48:47.5763743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3281
2022-11-23T01:48:47.5764404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5764861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5765416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5765881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5766449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5766885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5767427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5767886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5768333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5768812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5769462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5770147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5770662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5771109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5771441Z ok (4.330s)
2022-11-23T01:48:47.5771649Z 
2022-11-23T01:48:47.5771915Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5772218Z Ran 1 test in 4.330s
2022-11-23T01:48:47.5772380Z 
2022-11-23T01:48:47.5772473Z OK
2022-11-23T01:48:47.5772607Z 
2022-11-23T01:48:47.5772736Z Generating XML reports...
2022-11-23T01:48:47.5773341Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012439.xml
2022-11-23T01:48:47.5774032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5774471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5775036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5775500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5775708Z 
2022-11-23T01:48:47.5775823Z Running tests...
2022-11-23T01:48:47.5776223Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5776750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5777296Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5778670Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2022-11-23T01:48:47.5779194Z 
2022-11-23T01:48:47.5779464Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5779791Z Ran 1 test in 1.642s
2022-11-23T01:48:47.5779951Z 
2022-11-23T01:48:47.5780044Z OK (skipped=1)
2022-11-23T01:48:47.5780204Z 
2022-11-23T01:48:47.5780330Z Generating XML reports...
2022-11-23T01:48:47.5780934Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012446.xml
2022-11-23T01:48:47.5781872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5782320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5782895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5783363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5783590Z 
2022-11-23T01:48:47.5783702Z Running tests...
2022-11-23T01:48:47.5784084Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5784608Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5785141Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5785628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3419
2022-11-23T01:48:47.5786076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3420
2022-11-23T01:48:47.5786674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5787124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5787677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5788142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5788714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5789136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5789787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5790254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5790706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5791184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5791836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5792521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5793040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5793485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5793828Z ok (7.058s)
2022-11-23T01:48:47.5793977Z 
2022-11-23T01:48:47.5794243Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5794551Z Ran 1 test in 7.058s
2022-11-23T01:48:47.5794710Z 
2022-11-23T01:48:47.5794809Z OK
2022-11-23T01:48:47.5794944Z 
2022-11-23T01:48:47.5795070Z Generating XML reports...
2022-11-23T01:48:47.5795660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012450.xml
2022-11-23T01:48:47.5796372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5796820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5797390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5797839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5798071Z 
2022-11-23T01:48:47.5798181Z Running tests...
2022-11-23T01:48:47.5798588Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5799183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5799717Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T01:48:47.5800003Z 
2022-11-23T01:48:47.5800273Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5800595Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5800755Z 
2022-11-23T01:48:47.5800845Z OK (skipped=1)
2022-11-23T01:48:47.5801000Z 
2022-11-23T01:48:47.5801126Z Generating XML reports...
2022-11-23T01:48:47.5801725Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012459.xml
2022-11-23T01:48:47.5802431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5802864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5803439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5803906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5804134Z 
2022-11-23T01:48:47.5804224Z Running tests...
2022-11-23T01:48:47.5804628Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5805155Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5805608Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5806063Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5806537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3567
2022-11-23T01:48:47.5807047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3568
2022-11-23T01:48:47.5807633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5808084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5808650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5809116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5809670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5810116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5810674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5811142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5811570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5812073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5812721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5813393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5813910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5814377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5814875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo4hurlp
2022-11-23T01:48:47.5815399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo4hurlp/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5815923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7y3q7l4y
2022-11-23T01:48:47.5816512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7y3q7l4y/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5816904Z ok (7.458s)
2022-11-23T01:48:47.5817033Z 
2022-11-23T01:48:47.5817307Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5817636Z Ran 1 test in 7.458s
2022-11-23T01:48:47.5817799Z 
2022-11-23T01:48:47.5818151Z OK
2022-11-23T01:48:47.5818282Z 
2022-11-23T01:48:47.5818408Z Generating XML reports...
2022-11-23T01:48:47.5819022Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012502.xml
2022-11-23T01:48:47.5819733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5820187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5820748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5821220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5821448Z 
2022-11-23T01:48:47.5821559Z Running tests...
2022-11-23T01:48:47.5821944Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5822470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5822945Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5823436Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5823896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3686
2022-11-23T01:48:47.5824437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3687
2022-11-23T01:48:47.5825046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5825478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5826051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5826517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5827092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5827516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5828076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5828539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5828969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5829468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5830121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5830808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5831306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5831772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5832271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchzhl3w7
2022-11-23T01:48:47.5832814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchzhl3w7/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5833323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmh216w7n
2022-11-23T01:48:47.5833920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmh216w7n/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5834443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5834904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5835257Z ok (7.357s)
2022-11-23T01:48:47.5835407Z 
2022-11-23T01:48:47.5835680Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5836007Z Ran 1 test in 7.358s
2022-11-23T01:48:47.5836147Z 
2022-11-23T01:48:47.5836245Z OK
2022-11-23T01:48:47.5836379Z 
2022-11-23T01:48:47.5836505Z Generating XML reports...
2022-11-23T01:48:47.5837110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012512.xml
2022-11-23T01:48:47.5837813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5838264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5838834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5839298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5839505Z 
2022-11-23T01:48:47.5839620Z Running tests...
2022-11-23T01:48:47.5840023Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5840551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5841025Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5841626Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5842129Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3805
2022-11-23T01:48:47.5842578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3806
2022-11-23T01:48:47.5843164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5843612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5844184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5844652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5845203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5845641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5846213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5846656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5847108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5847605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5848258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5848926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5849444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5849911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5850411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2xj6hg6
2022-11-23T01:48:47.5850983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2xj6hg6/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5851519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdt1cd0h
2022-11-23T01:48:47.5852103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdt1cd0h/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5852597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5853074Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.5853424Z ok (7.352s)
2022-11-23T01:48:47.5853572Z 
2022-11-23T01:48:47.5853845Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5854154Z Ran 1 test in 7.352s
2022-11-23T01:48:47.5854318Z 
2022-11-23T01:48:47.5854414Z OK
2022-11-23T01:48:47.5854549Z 
2022-11-23T01:48:47.5854673Z Generating XML reports...
2022-11-23T01:48:47.5855259Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012522.xml
2022-11-23T01:48:47.5855976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5856425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5856998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5857447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5857674Z 
2022-11-23T01:48:47.5857787Z Running tests...
2022-11-23T01:48:47.5858438Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5858952Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5859519Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:47.5860010Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5860491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3924
2022-11-23T01:48:47.5860916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3925
2022-11-23T01:48:47.5861518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5861964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5862515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5862981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5863558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5863993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5864542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5865003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5865452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5865931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5866585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5867269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5867797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5868245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5868806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0x2wnace
2022-11-23T01:48:47.5869350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0x2wnace/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5869873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6qvyi83
2022-11-23T01:48:47.5870385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6qvyi83/_remote_module_non_scriptable.py
2022-11-23T01:48:47.5870761Z ok (7.348s)
2022-11-23T01:48:47.5870910Z 
2022-11-23T01:48:47.5871185Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5871494Z Ran 1 test in 7.348s
2022-11-23T01:48:47.5871660Z 
2022-11-23T01:48:47.5871756Z OK
2022-11-23T01:48:47.5871892Z 
2022-11-23T01:48:47.5872019Z Generating XML reports...
2022-11-23T01:48:47.5872624Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012532.xml
2022-11-23T01:48:47.5873320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5873765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5874335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5874778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5875003Z 
2022-11-23T01:48:47.5875114Z Running tests...
2022-11-23T01:48:47.5875517Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5876040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5876580Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.5876862Z 
2022-11-23T01:48:47.5877127Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5877460Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5877619Z 
2022-11-23T01:48:47.5877711Z OK (skipped=1)
2022-11-23T01:48:47.5877870Z 
2022-11-23T01:48:47.5877996Z Generating XML reports...
2022-11-23T01:48:47.5878597Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012541.xml
2022-11-23T01:48:47.5879306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5879732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5880302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5880773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5880997Z 
2022-11-23T01:48:47.5881109Z Running tests...
2022-11-23T01:48:47.5881493Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5882026Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5882560Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T01:48:47.5882880Z 
2022-11-23T01:48:47.5883124Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5883449Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5883607Z 
2022-11-23T01:48:47.5883721Z OK (skipped=1)
2022-11-23T01:48:47.5883875Z 
2022-11-23T01:48:47.5883999Z Generating XML reports...
2022-11-23T01:48:47.5884580Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012544.xml
2022-11-23T01:48:47.5885294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5885740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5886356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5886838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5887065Z 
2022-11-23T01:48:47.5887176Z Running tests...
2022-11-23T01:48:47.5887580Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5888088Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5888620Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T01:48:47.5888944Z 
2022-11-23T01:48:47.5889205Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5889531Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5889671Z 
2022-11-23T01:48:47.5889782Z OK (skipped=1)
2022-11-23T01:48:47.5889936Z 
2022-11-23T01:48:47.5890061Z Generating XML reports...
2022-11-23T01:48:47.5890663Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012546.xml
2022-11-23T01:48:47.5891351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5891793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5892362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5892823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5893030Z 
2022-11-23T01:48:47.5893198Z Running tests...
2022-11-23T01:48:47.5893600Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5894120Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5894633Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T01:48:47.5894948Z 
2022-11-23T01:48:47.5895210Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5895531Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5895690Z 
2022-11-23T01:48:47.5895798Z OK (skipped=1)
2022-11-23T01:48:47.5895933Z 
2022-11-23T01:48:47.5896054Z Generating XML reports...
2022-11-23T01:48:47.5896686Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012548.xml
2022-11-23T01:48:47.5897393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5897843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5898646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5899116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5899345Z 
2022-11-23T01:48:47.5899452Z Running tests...
2022-11-23T01:48:47.5899868Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5900394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5900926Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T01:48:47.5901233Z 
2022-11-23T01:48:47.5901495Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5901801Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5901962Z 
2022-11-23T01:48:47.5902072Z OK (skipped=1)
2022-11-23T01:48:47.5902091Z 
2022-11-23T01:48:47.5902214Z Generating XML reports...
2022-11-23T01:48:47.5902660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012551.xml
2022-11-23T01:48:47.5903106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5903289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5903655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5903846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5903866Z 
2022-11-23T01:48:47.5903975Z Running tests...
2022-11-23T01:48:47.5904236Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5904547Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5904843Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-11-23T01:48:47.5904864Z 
2022-11-23T01:48:47.5905129Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5905240Z Ran 1 test in 0.003s
2022-11-23T01:48:47.5905259Z 
2022-11-23T01:48:47.5905365Z OK (skipped=1)
2022-11-23T01:48:47.5905385Z 
2022-11-23T01:48:47.5905491Z Generating XML reports...
2022-11-23T01:48:47.5905937Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012553.xml
2022-11-23T01:48:47.5906304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5906481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5906857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5907114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5907134Z 
2022-11-23T01:48:47.5907242Z Running tests...
2022-11-23T01:48:47.5907512Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5907810Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5908075Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.5908095Z 
2022-11-23T01:48:47.5908352Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5908464Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5908483Z 
2022-11-23T01:48:47.5908591Z OK (skipped=1)
2022-11-23T01:48:47.5908610Z 
2022-11-23T01:48:47.5908732Z Generating XML reports...
2022-11-23T01:48:47.5909177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012556.xml
2022-11-23T01:48:47.5909550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5909729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5910085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5910271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5910290Z 
2022-11-23T01:48:47.5910398Z Running tests...
2022-11-23T01:48:47.5910659Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5910970Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5911228Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5911448Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4274
2022-11-23T01:48:47.5911662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4275
2022-11-23T01:48:47.5912064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5912246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5912628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5912817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5913178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5913350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5913722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5913912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5914152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5914380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5914776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5915170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5915400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5915624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5915955Z STAGE:2022-11-23 01:26:04 4275:4275 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5916340Z STAGE:2022-11-23 01:26:04 4274:4274 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5916672Z STAGE:2022-11-23 01:26:05 4274:4274 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5916998Z STAGE:2022-11-23 01:26:05 4275:4275 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5917322Z STAGE:2022-11-23 01:26:05 4274:4274 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5917656Z STAGE:2022-11-23 01:26:05 4275:4275 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5917972Z STAGE:2022-11-23 01:26:05 4275:4275 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5918296Z STAGE:2022-11-23 01:26:05 4275:4275 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5918631Z STAGE:2022-11-23 01:26:05 4275:4275 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5918950Z STAGE:2022-11-23 01:26:05 4274:4274 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5919282Z STAGE:2022-11-23 01:26:05 4274:4274 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5919623Z STAGE:2022-11-23 01:26:05 4274:4274 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5919709Z ok (7.751s)
2022-11-23T01:48:47.5919746Z 
2022-11-23T01:48:47.5919995Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5920106Z Ran 1 test in 7.751s
2022-11-23T01:48:47.5920126Z 
2022-11-23T01:48:47.5920217Z OK
2022-11-23T01:48:47.5920237Z 
2022-11-23T01:48:47.5920359Z Generating XML reports...
2022-11-23T01:48:47.5920809Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012558.xml
2022-11-23T01:48:47.5921184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5921358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5921792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5921971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5922006Z 
2022-11-23T01:48:47.5922097Z Running tests...
2022-11-23T01:48:47.5922361Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5922676Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5922943Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5923160Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4393
2022-11-23T01:48:47.5923377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4394
2022-11-23T01:48:47.5923747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5923922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5924284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5924473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5924833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5925005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5925375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5925560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5925855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5926100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5926484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5926883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5927109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5927335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5927667Z STAGE:2022-11-23 01:26:14 4394:4394 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5927992Z STAGE:2022-11-23 01:26:14 4393:4393 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5928329Z STAGE:2022-11-23 01:26:15 4393:4393 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5928652Z STAGE:2022-11-23 01:26:15 4394:4394 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5928998Z STAGE:2022-11-23 01:26:15 4393:4393 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5929322Z STAGE:2022-11-23 01:26:15 4394:4394 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5929640Z STAGE:2022-11-23 01:26:15 4394:4394 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5929963Z STAGE:2022-11-23 01:26:15 4394:4394 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5930302Z STAGE:2022-11-23 01:26:15 4394:4394 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5930623Z STAGE:2022-11-23 01:26:15 4393:4393 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5930948Z STAGE:2022-11-23 01:26:15 4393:4393 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5931332Z STAGE:2022-11-23 01:26:15 4393:4393 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5931442Z ok (7.747s)
2022-11-23T01:48:47.5931462Z 
2022-11-23T01:48:47.5931724Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5931817Z Ran 1 test in 7.747s
2022-11-23T01:48:47.5931837Z 
2022-11-23T01:48:47.5931932Z OK
2022-11-23T01:48:47.5931951Z 
2022-11-23T01:48:47.5932075Z Generating XML reports...
2022-11-23T01:48:47.5932529Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012608.xml
2022-11-23T01:48:47.5932897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5933077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5933454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5933647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5933667Z 
2022-11-23T01:48:47.5933758Z Running tests...
2022-11-23T01:48:47.5934020Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5934333Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5934602Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.5934621Z 
2022-11-23T01:48:47.5934882Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5934992Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5935011Z 
2022-11-23T01:48:47.5935119Z OK (skipped=1)
2022-11-23T01:48:47.5935190Z 
2022-11-23T01:48:47.5935316Z Generating XML reports...
2022-11-23T01:48:47.5935761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012619.xml
2022-11-23T01:48:47.5936114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5936290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5936665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5936853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5936872Z 
2022-11-23T01:48:47.5936980Z Running tests...
2022-11-23T01:48:47.5937239Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5937552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5937818Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.5937838Z 
2022-11-23T01:48:47.5938341Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5938442Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5938462Z 
2022-11-23T01:48:47.5938570Z OK (skipped=1)
2022-11-23T01:48:47.5938590Z 
2022-11-23T01:48:47.5938714Z Generating XML reports...
2022-11-23T01:48:47.5939162Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012621.xml
2022-11-23T01:48:47.5939533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5939708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5940082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5940274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5940294Z 
2022-11-23T01:48:47.5940400Z Running tests...
2022-11-23T01:48:47.5940642Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5941023Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5941307Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5941522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4578
2022-11-23T01:48:47.5941733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4579
2022-11-23T01:48:47.5942102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5942278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5942656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5942826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5943191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5943362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5943733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5943920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5944161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5944403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5944804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5945274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5945489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5945714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5946043Z STAGE:2022-11-23 01:26:29 4579:4579 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5946370Z STAGE:2022-11-23 01:26:29 4578:4578 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5946701Z STAGE:2022-11-23 01:26:30 4578:4578 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5947025Z STAGE:2022-11-23 01:26:30 4579:4579 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5947372Z STAGE:2022-11-23 01:26:30 4578:4578 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5947707Z STAGE:2022-11-23 01:26:30 4579:4579 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5947792Z ok (7.644s)
2022-11-23T01:48:47.5947831Z 
2022-11-23T01:48:47.5948076Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5948188Z Ran 1 test in 7.644s
2022-11-23T01:48:47.5948208Z 
2022-11-23T01:48:47.5948301Z OK
2022-11-23T01:48:47.5948319Z 
2022-11-23T01:48:47.5948442Z Generating XML reports...
2022-11-23T01:48:47.5948890Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012623.xml
2022-11-23T01:48:47.5949259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5949434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5949817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5949987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5950007Z 
2022-11-23T01:48:47.5950115Z Running tests...
2022-11-23T01:48:47.5950429Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5950755Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5951030Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5951244Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4697
2022-11-23T01:48:47.5951456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4698
2022-11-23T01:48:47.5951825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5951986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5952361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5952552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5952915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5953086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5953464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5953648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5953888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5954129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5954570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5954968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5955195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5955419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5955753Z STAGE:2022-11-23 01:26:39 4697:4697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5956077Z STAGE:2022-11-23 01:26:39 4698:4698 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.5956409Z STAGE:2022-11-23 01:26:41 4697:4697 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5956746Z STAGE:2022-11-23 01:26:41 4698:4698 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.5957088Z STAGE:2022-11-23 01:26:41 4697:4697 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5957414Z STAGE:2022-11-23 01:26:41 4698:4698 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.5957518Z ok (7.853s)
2022-11-23T01:48:47.5957539Z 
2022-11-23T01:48:47.5957802Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5957915Z Ran 1 test in 7.854s
2022-11-23T01:48:47.5957934Z 
2022-11-23T01:48:47.5958026Z OK
2022-11-23T01:48:47.5958045Z 
2022-11-23T01:48:47.5958167Z Generating XML reports...
2022-11-23T01:48:47.5958616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012634.xml
2022-11-23T01:48:47.5958984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5959145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5959523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5959775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5959796Z 
2022-11-23T01:48:47.5959908Z Running tests...
2022-11-23T01:48:47.5960172Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5960487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5960746Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5961497Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.650s)
2022-11-23T01:48:47.5961522Z 
2022-11-23T01:48:47.5961786Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5961898Z Ran 1 test in 1.651s
2022-11-23T01:48:47.5961920Z 
2022-11-23T01:48:47.5962012Z OK (skipped=1)
2022-11-23T01:48:47.5962031Z 
2022-11-23T01:48:47.5962155Z Generating XML reports...
2022-11-23T01:48:47.5962601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012644.xml
2022-11-23T01:48:47.5962969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5963142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5963521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5963767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5963787Z 
2022-11-23T01:48:47.5963894Z Running tests...
2022-11-23T01:48:47.5964143Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5964461Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5964731Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5965470Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.626s)
2022-11-23T01:48:47.5965491Z 
2022-11-23T01:48:47.5965749Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5965866Z Ran 1 test in 1.626s
2022-11-23T01:48:47.5965888Z 
2022-11-23T01:48:47.5965997Z OK (skipped=1)
2022-11-23T01:48:47.5966016Z 
2022-11-23T01:48:47.5966144Z Generating XML reports...
2022-11-23T01:48:47.5966591Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012648.xml
2022-11-23T01:48:47.5966961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5967119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5967496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5967683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5967703Z 
2022-11-23T01:48:47.5967812Z Running tests...
2022-11-23T01:48:47.5968074Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5968386Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5968663Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5968926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4884
2022-11-23T01:48:47.5969147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4885
2022-11-23T01:48:47.5969499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5969672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5970045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5970234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5970597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5970771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5971145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5971334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5971560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5971800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5972199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5972593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5972821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5973099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5973203Z ok (6.830s)
2022-11-23T01:48:47.5973223Z 
2022-11-23T01:48:47.5973491Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5973602Z Ran 1 test in 6.830s
2022-11-23T01:48:47.5973622Z 
2022-11-23T01:48:47.5973696Z OK
2022-11-23T01:48:47.5973715Z 
2022-11-23T01:48:47.5973839Z Generating XML reports...
2022-11-23T01:48:47.5974287Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012652.xml
2022-11-23T01:48:47.5974772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5974946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5975323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5975517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5975537Z 
2022-11-23T01:48:47.5975646Z Running tests...
2022-11-23T01:48:47.5975891Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5976204Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5976470Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5976685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4999
2022-11-23T01:48:47.5976897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5000
2022-11-23T01:48:47.5977262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5977439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5977820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5978232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5978663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5978845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5979219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5979407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5979649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5979891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5980291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5980684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5980917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5981124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5981357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.5981595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.5981997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.5982387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.5982728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:47.5982965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:47.5983354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.5983747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.5983968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:48:47.5984203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:48:47.5984588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T01:48:47.5984981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T01:48:47.5985085Z ok (8.248s)
2022-11-23T01:48:47.5985104Z 
2022-11-23T01:48:47.5985369Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5985482Z Ran 1 test in 8.248s
2022-11-23T01:48:47.5985501Z 
2022-11-23T01:48:47.5985593Z OK
2022-11-23T01:48:47.5985612Z 
2022-11-23T01:48:47.5985734Z Generating XML reports...
2022-11-23T01:48:47.5986164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012702.xml
2022-11-23T01:48:47.5986534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5986709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5987087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5987280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5987300Z 
2022-11-23T01:48:47.5987410Z Running tests...
2022-11-23T01:48:47.5987719Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5988044Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5988282Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.5988497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5150
2022-11-23T01:48:47.5988709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5151
2022-11-23T01:48:47.5989079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5989256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5989640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5989829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5990192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5990362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5990726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5990912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5991153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.5991392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.5991848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5992239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.5992470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.5992695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.5992779Z ok (6.803s)
2022-11-23T01:48:47.5992819Z 
2022-11-23T01:48:47.5993063Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5993175Z Ran 1 test in 6.804s
2022-11-23T01:48:47.5993195Z 
2022-11-23T01:48:47.5993285Z OK
2022-11-23T01:48:47.5993304Z 
2022-11-23T01:48:47.5993426Z Generating XML reports...
2022-11-23T01:48:47.5993873Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012712.xml
2022-11-23T01:48:47.5994243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5994419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5994800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5994970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5995007Z 
2022-11-23T01:48:47.5995099Z Running tests...
2022-11-23T01:48:47.5995360Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5995675Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5996093Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.5996117Z 
2022-11-23T01:48:47.5996380Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5996493Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5996513Z 
2022-11-23T01:48:47.5996621Z OK (skipped=1)
2022-11-23T01:48:47.5996640Z 
2022-11-23T01:48:47.5996828Z Generating XML reports...
2022-11-23T01:48:47.5997272Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012722.xml
2022-11-23T01:48:47.5997639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.5997811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.5998189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.5998379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.5998398Z 
2022-11-23T01:48:47.5998511Z Running tests...
2022-11-23T01:48:47.5998774Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5999089Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.5999491Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.5999531Z 
2022-11-23T01:48:47.5999818Z ----------------------------------------------------------------------
2022-11-23T01:48:47.5999932Z Ran 1 test in 0.002s
2022-11-23T01:48:47.5999951Z 
2022-11-23T01:48:47.6000058Z OK (skipped=1)
2022-11-23T01:48:47.6000077Z 
2022-11-23T01:48:47.6000201Z Generating XML reports...
2022-11-23T01:48:47.6000648Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012724.xml
2022-11-23T01:48:47.6001014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6001243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6001627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6001802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6001840Z 
2022-11-23T01:48:47.6001930Z Running tests...
2022-11-23T01:48:47.6002189Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6002501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6002926Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6002948Z 
2022-11-23T01:48:47.6003204Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6003317Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6003340Z 
2022-11-23T01:48:47.6003446Z OK (skipped=1)
2022-11-23T01:48:47.6003465Z 
2022-11-23T01:48:47.6003587Z Generating XML reports...
2022-11-23T01:48:47.6004017Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012726.xml
2022-11-23T01:48:47.6004392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6004568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6004944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6005132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6005151Z 
2022-11-23T01:48:47.6005260Z Running tests...
2022-11-23T01:48:47.6005524Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6005832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6006249Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6006270Z 
2022-11-23T01:48:47.6006565Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6006686Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6006705Z 
2022-11-23T01:48:47.6006815Z OK (skipped=1)
2022-11-23T01:48:47.6006834Z 
2022-11-23T01:48:47.6006959Z Generating XML reports...
2022-11-23T01:48:47.6007409Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012729.xml
2022-11-23T01:48:47.6007780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6007954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6008328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6008521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6008540Z 
2022-11-23T01:48:47.6008631Z Running tests...
2022-11-23T01:48:47.6008943Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6009265Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6009673Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6009693Z 
2022-11-23T01:48:47.6009952Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6010063Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6010082Z 
2022-11-23T01:48:47.6010188Z OK (skipped=1)
2022-11-23T01:48:47.6010207Z 
2022-11-23T01:48:47.6010330Z Generating XML reports...
2022-11-23T01:48:47.6010840Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012731.xml
2022-11-23T01:48:47.6011192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6011367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6011744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6011931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6011951Z 
2022-11-23T01:48:47.6012059Z Running tests...
2022-11-23T01:48:47.6012323Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6012639Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6013045Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6013070Z 
2022-11-23T01:48:47.6013327Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6013422Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6013441Z 
2022-11-23T01:48:47.6013553Z OK (skipped=1)
2022-11-23T01:48:47.6013573Z 
2022-11-23T01:48:47.6013697Z Generating XML reports...
2022-11-23T01:48:47.6014143Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012734.xml
2022-11-23T01:48:47.6014511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6014682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6015059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6015248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6015271Z 
2022-11-23T01:48:47.6015361Z Running tests...
2022-11-23T01:48:47.6015621Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6015986Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6016413Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6016433Z 
2022-11-23T01:48:47.6016690Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6016801Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6016820Z 
2022-11-23T01:48:47.6016928Z OK (skipped=1)
2022-11-23T01:48:47.6016946Z 
2022-11-23T01:48:47.6017070Z Generating XML reports...
2022-11-23T01:48:47.6017514Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012736.xml
2022-11-23T01:48:47.6017869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6018304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6018697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6018888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6018907Z 
2022-11-23T01:48:47.6019014Z Running tests...
2022-11-23T01:48:47.6019275Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6019588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6019996Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6020016Z 
2022-11-23T01:48:47.6020277Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6020454Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6020474Z 
2022-11-23T01:48:47.6020581Z OK (skipped=1)
2022-11-23T01:48:47.6020600Z 
2022-11-23T01:48:47.6020723Z Generating XML reports...
2022-11-23T01:48:47.6021179Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012738.xml
2022-11-23T01:48:47.6021548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6021720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6022097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6022288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6022308Z 
2022-11-23T01:48:47.6022415Z Running tests...
2022-11-23T01:48:47.6022660Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6022975Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6023371Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6023395Z 
2022-11-23T01:48:47.6023653Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6023766Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6023785Z 
2022-11-23T01:48:47.6023891Z OK (skipped=1)
2022-11-23T01:48:47.6023910Z 
2022-11-23T01:48:47.6024033Z Generating XML reports...
2022-11-23T01:48:47.6024475Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012741.xml
2022-11-23T01:48:47.6024824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6024999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6025380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6025566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6025585Z 
2022-11-23T01:48:47.6025767Z Running tests...
2022-11-23T01:48:47.6026042Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6026353Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6026654Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6026674Z 
2022-11-23T01:48:47.6026932Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6027027Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6027063Z 
2022-11-23T01:48:47.6027153Z OK (skipped=1)
2022-11-23T01:48:47.6027176Z 
2022-11-23T01:48:47.6027297Z Generating XML reports...
2022-11-23T01:48:47.6027743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012743.xml
2022-11-23T01:48:47.6028112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6028286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6028662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6028851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6028870Z 
2022-11-23T01:48:47.6028978Z Running tests...
2022-11-23T01:48:47.6029219Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6029530Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6029985Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6030006Z 
2022-11-23T01:48:47.6030266Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6030377Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6030400Z 
2022-11-23T01:48:47.6030508Z OK (skipped=1)
2022-11-23T01:48:47.6030527Z 
2022-11-23T01:48:47.6030652Z Generating XML reports...
2022-11-23T01:48:47.6031096Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012746.xml
2022-11-23T01:48:47.6031463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6031618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6031989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6032182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6032201Z 
2022-11-23T01:48:47.6032308Z Running tests...
2022-11-23T01:48:47.6032569Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6032884Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6033290Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6033310Z 
2022-11-23T01:48:47.6033570Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6033665Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6033701Z 
2022-11-23T01:48:47.6033792Z OK (skipped=1)
2022-11-23T01:48:47.6033811Z 
2022-11-23T01:48:47.6033932Z Generating XML reports...
2022-11-23T01:48:47.6034379Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012748.xml
2022-11-23T01:48:47.6034753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6034927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6035354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6035552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6035573Z 
2022-11-23T01:48:47.6035682Z Running tests...
2022-11-23T01:48:47.6035929Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6036244Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6036639Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.6036660Z 
2022-11-23T01:48:47.6036923Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6037033Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6037051Z 
2022-11-23T01:48:47.6037157Z OK (skipped=1)
2022-11-23T01:48:47.6037176Z 
2022-11-23T01:48:47.6037299Z Generating XML reports...
2022-11-23T01:48:47.6037742Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012750.xml
2022-11-23T01:48:47.6038113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6038268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6038642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6038829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6038848Z 
2022-11-23T01:48:47.6038957Z Running tests...
2022-11-23T01:48:47.6039273Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6039587Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6039878Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6039897Z 
2022-11-23T01:48:47.6040156Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6040269Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6040288Z 
2022-11-23T01:48:47.6040377Z OK (skipped=1)
2022-11-23T01:48:47.6040396Z 
2022-11-23T01:48:47.6040519Z Generating XML reports...
2022-11-23T01:48:47.6040964Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012753.xml
2022-11-23T01:48:47.6041329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6041505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6041885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6042071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6042094Z 
2022-11-23T01:48:47.6042200Z Running tests...
2022-11-23T01:48:47.6042443Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6042757Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6043026Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6043046Z 
2022-11-23T01:48:47.6043311Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6043423Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6043442Z 
2022-11-23T01:48:47.6043550Z OK (skipped=1)
2022-11-23T01:48:47.6043572Z 
2022-11-23T01:48:47.6043696Z Generating XML reports...
2022-11-23T01:48:47.6044140Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012755.xml
2022-11-23T01:48:47.6044557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6044721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6045099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6045288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6045308Z 
2022-11-23T01:48:47.6045416Z Running tests...
2022-11-23T01:48:47.6045677Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6045987Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6046264Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6046284Z 
2022-11-23T01:48:47.6046540Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6046655Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6046674Z 
2022-11-23T01:48:47.6046764Z OK (skipped=1)
2022-11-23T01:48:47.6046783Z 
2022-11-23T01:48:47.6046906Z Generating XML reports...
2022-11-23T01:48:47.6047350Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012757.xml
2022-11-23T01:48:47.6047718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6047892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6048269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6048518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6048538Z 
2022-11-23T01:48:47.6048649Z Running tests...
2022-11-23T01:48:47.6048909Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6049205Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6049482Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6049502Z 
2022-11-23T01:48:47.6049762Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6049873Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6049892Z 
2022-11-23T01:48:47.6049998Z OK (skipped=1)
2022-11-23T01:48:47.6050016Z 
2022-11-23T01:48:47.6050138Z Generating XML reports...
2022-11-23T01:48:47.6050584Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012800.xml
2022-11-23T01:48:47.6050957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6051133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6051497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6051684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6051704Z 
2022-11-23T01:48:47.6051811Z Running tests...
2022-11-23T01:48:47.6052073Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6052383Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6052652Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6052672Z 
2022-11-23T01:48:47.6052935Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6053047Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6053066Z 
2022-11-23T01:48:47.6053155Z OK (skipped=1)
2022-11-23T01:48:47.6053191Z 
2022-11-23T01:48:47.6053295Z Generating XML reports...
2022-11-23T01:48:47.6053787Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012802.xml
2022-11-23T01:48:47.6054169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6054343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6054721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6054911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6054931Z 
2022-11-23T01:48:47.6055041Z Running tests...
2022-11-23T01:48:47.6055306Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6055601Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6055868Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6055888Z 
2022-11-23T01:48:47.6056147Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6056261Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6056280Z 
2022-11-23T01:48:47.6056386Z OK (skipped=1)
2022-11-23T01:48:47.6056405Z 
2022-11-23T01:48:47.6056527Z Generating XML reports...
2022-11-23T01:48:47.6056970Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012805.xml
2022-11-23T01:48:47.6057337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6057508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6058228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6058433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6058457Z 
2022-11-23T01:48:47.6058566Z Running tests...
2022-11-23T01:48:47.6058830Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6059140Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6059403Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6059422Z 
2022-11-23T01:48:47.6059680Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6059792Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6059812Z 
2022-11-23T01:48:47.6059901Z OK (skipped=1)
2022-11-23T01:48:47.6059941Z 
2022-11-23T01:48:47.6060048Z Generating XML reports...
2022-11-23T01:48:47.6060488Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012807.xml
2022-11-23T01:48:47.6060857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6061031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6061407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6061593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6061612Z 
2022-11-23T01:48:47.6061719Z Running tests...
2022-11-23T01:48:47.6061977Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6062269Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6062543Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6062563Z 
2022-11-23T01:48:47.6062821Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6062932Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6063031Z 
2022-11-23T01:48:47.6063148Z OK (skipped=1)
2022-11-23T01:48:47.6063167Z 
2022-11-23T01:48:47.6063291Z Generating XML reports...
2022-11-23T01:48:47.6063743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012809.xml
2022-11-23T01:48:47.6064113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6064286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6064643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6064837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6064857Z 
2022-11-23T01:48:47.6064965Z Running tests...
2022-11-23T01:48:47.6065227Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6065544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6065808Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6065827Z 
2022-11-23T01:48:47.6066086Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6066197Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6066216Z 
2022-11-23T01:48:47.6066324Z OK (skipped=1)
2022-11-23T01:48:47.6066345Z 
2022-11-23T01:48:47.6066449Z Generating XML reports...
2022-11-23T01:48:47.6066891Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012812.xml
2022-11-23T01:48:47.6067333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6067506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6067884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6068074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6068093Z 
2022-11-23T01:48:47.6068201Z Running tests...
2022-11-23T01:48:47.6068459Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6068753Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6069012Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6069032Z 
2022-11-23T01:48:47.6069285Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6069400Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6069420Z 
2022-11-23T01:48:47.6069526Z OK (skipped=1)
2022-11-23T01:48:47.6069545Z 
2022-11-23T01:48:47.6069668Z Generating XML reports...
2022-11-23T01:48:47.6070115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012814.xml
2022-11-23T01:48:47.6070483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6070656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6071015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6071202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6071222Z 
2022-11-23T01:48:47.6071330Z Running tests...
2022-11-23T01:48:47.6071589Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6071903Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6072155Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6072222Z 
2022-11-23T01:48:47.6072488Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6072600Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6072620Z 
2022-11-23T01:48:47.6072728Z OK (skipped=1)
2022-11-23T01:48:47.6072747Z 
2022-11-23T01:48:47.6072852Z Generating XML reports...
2022-11-23T01:48:47.6073299Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012816.xml
2022-11-23T01:48:47.6073666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6073839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6074219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6074409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6074428Z 
2022-11-23T01:48:47.6074539Z Running tests...
2022-11-23T01:48:47.6074803Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6075098Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6075375Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-11-23T01:48:47.6075394Z 
2022-11-23T01:48:47.6075655Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6075766Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6075786Z 
2022-11-23T01:48:47.6075892Z OK (skipped=1)
2022-11-23T01:48:47.6075911Z 
2022-11-23T01:48:47.6076032Z Generating XML reports...
2022-11-23T01:48:47.6076538Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012819.xml
2022-11-23T01:48:47.6076910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6077084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6077442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6077630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6077649Z 
2022-11-23T01:48:47.6077756Z Running tests...
2022-11-23T01:48:47.6078014Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6078327Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6078615Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-11-23T01:48:47.6078638Z 
2022-11-23T01:48:47.6078898Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6079010Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6079029Z 
2022-11-23T01:48:47.6079139Z OK (skipped=1)
2022-11-23T01:48:47.6079158Z 
2022-11-23T01:48:47.6079262Z Generating XML reports...
2022-11-23T01:48:47.6079703Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012821.xml
2022-11-23T01:48:47.6080067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6080241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6080612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6080800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6080822Z 
2022-11-23T01:48:47.6080928Z Running tests...
2022-11-23T01:48:47.6081188Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6081551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6081806Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6081827Z 
2022-11-23T01:48:47.6082087Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6082198Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6082217Z 
2022-11-23T01:48:47.6082323Z OK (skipped=1)
2022-11-23T01:48:47.6082342Z 
2022-11-23T01:48:47.6082464Z Generating XML reports...
2022-11-23T01:48:47.6082900Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012824.xml
2022-11-23T01:48:47.6083269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6083441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6083800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6083988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6084008Z 
2022-11-23T01:48:47.6084114Z Running tests...
2022-11-23T01:48:47.6084372Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6084683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6084947Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6085165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6156
2022-11-23T01:48:47.6085460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6157
2022-11-23T01:48:47.6085829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6085985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6086363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6086552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6086917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6087088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6087460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6087646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6087892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6088114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6088514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6088903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6089128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6089352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6089454Z ok (6.942s)
2022-11-23T01:48:47.6089473Z 
2022-11-23T01:48:47.6089736Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6089852Z Ran 1 test in 6.943s
2022-11-23T01:48:47.6089871Z 
2022-11-23T01:48:47.6089963Z OK
2022-11-23T01:48:47.6089982Z 
2022-11-23T01:48:47.6090088Z Generating XML reports...
2022-11-23T01:48:47.6090600Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012826.xml
2022-11-23T01:48:47.6090982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6091161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6091536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6091724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6091743Z 
2022-11-23T01:48:47.6091852Z Running tests...
2022-11-23T01:48:47.6092112Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6092428Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6092668Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6092688Z 
2022-11-23T01:48:47.6092945Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6093056Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6093076Z 
2022-11-23T01:48:47.6093183Z OK (skipped=1)
2022-11-23T01:48:47.6093203Z 
2022-11-23T01:48:47.6093326Z Generating XML reports...
2022-11-23T01:48:47.6093772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012835.xml
2022-11-23T01:48:47.6094139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6094314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6094675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6094919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6094938Z 
2022-11-23T01:48:47.6095046Z Running tests...
2022-11-23T01:48:47.6095314Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6095628Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6095894Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6095914Z 
2022-11-23T01:48:47.6096171Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6096283Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6096302Z 
2022-11-23T01:48:47.6096409Z OK (skipped=1)
2022-11-23T01:48:47.6096428Z 
2022-11-23T01:48:47.6096534Z Generating XML reports...
2022-11-23T01:48:47.6096976Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012838.xml
2022-11-23T01:48:47.6097345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6097526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6098105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6098301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6098321Z 
2022-11-23T01:48:47.6098429Z Running tests...
2022-11-23T01:48:47.6098693Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6099001Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6099252Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6099276Z 
2022-11-23T01:48:47.6099539Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6099680Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6099701Z 
2022-11-23T01:48:47.6099808Z OK (skipped=1)
2022-11-23T01:48:47.6099827Z 
2022-11-23T01:48:47.6100032Z Generating XML reports...
2022-11-23T01:48:47.6100494Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012840.xml
2022-11-23T01:48:47.6100860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6101033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6101407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6101579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6101603Z 
2022-11-23T01:48:47.6101710Z Running tests...
2022-11-23T01:48:47.6101971Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6102282Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6102542Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6102756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6370
2022-11-23T01:48:47.6102967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6371
2022-11-23T01:48:47.6103334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6103490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6103862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6104125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6104489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6104664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6105038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6105225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6105469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6105708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6106090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6106488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6106716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6106946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6107276Z STAGE:2022-11-23 01:28:48 6370:6370 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6107602Z STAGE:2022-11-23 01:28:48 6371:6371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6107938Z STAGE:2022-11-23 01:28:48 6371:6371 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6108265Z STAGE:2022-11-23 01:28:48 6370:6370 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6108606Z STAGE:2022-11-23 01:28:48 6371:6371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6108936Z STAGE:2022-11-23 01:28:48 6370:6370 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6109253Z STAGE:2022-11-23 01:28:48 6370:6370 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6109620Z STAGE:2022-11-23 01:28:48 6371:6371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6109957Z STAGE:2022-11-23 01:28:48 6370:6370 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6110297Z STAGE:2022-11-23 01:28:48 6370:6370 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6110629Z STAGE:2022-11-23 01:28:48 6371:6371 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6110968Z STAGE:2022-11-23 01:28:48 6371:6371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6111286Z STAGE:2022-11-23 01:28:48 6370:6370 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6111591Z STAGE:2022-11-23 01:28:48 6371:6371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6111916Z STAGE:2022-11-23 01:28:48 6370:6370 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6112254Z STAGE:2022-11-23 01:28:48 6370:6370 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6112582Z STAGE:2022-11-23 01:28:48 6371:6371 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6112914Z STAGE:2022-11-23 01:28:48 6371:6371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6113018Z ok (6.351s)
2022-11-23T01:48:47.6113038Z 
2022-11-23T01:48:47.6113299Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6113412Z Ran 1 test in 6.351s
2022-11-23T01:48:47.6113432Z 
2022-11-23T01:48:47.6113523Z OK
2022-11-23T01:48:47.6113542Z 
2022-11-23T01:48:47.6113648Z Generating XML reports...
2022-11-23T01:48:47.6114155Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012843.xml
2022-11-23T01:48:47.6114525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6114705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6115081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6115269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6115288Z 
2022-11-23T01:48:47.6115397Z Running tests...
2022-11-23T01:48:47.6115663Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6115957Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6116222Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6116442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6488
2022-11-23T01:48:47.6116656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6489
2022-11-23T01:48:47.6117026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6117199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6117575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6117761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6118118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6118270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6118652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6118834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6119121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6119371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6119772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6120166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6120395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6120619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6120937Z STAGE:2022-11-23 01:28:57 6488:6488 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6121261Z STAGE:2022-11-23 01:28:57 6489:6489 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6121592Z STAGE:2022-11-23 01:28:57 6488:6488 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6121921Z STAGE:2022-11-23 01:28:57 6489:6489 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6122265Z STAGE:2022-11-23 01:28:57 6488:6488 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6122604Z STAGE:2022-11-23 01:28:57 6489:6489 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6122928Z STAGE:2022-11-23 01:28:57 6488:6488 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6123250Z STAGE:2022-11-23 01:28:57 6489:6489 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6123637Z STAGE:2022-11-23 01:28:57 6488:6488 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6123963Z STAGE:2022-11-23 01:28:57 6488:6488 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6124285Z STAGE:2022-11-23 01:28:57 6489:6489 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6124622Z STAGE:2022-11-23 01:28:57 6489:6489 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6124942Z STAGE:2022-11-23 01:28:57 6488:6488 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6125265Z STAGE:2022-11-23 01:28:57 6489:6489 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6125594Z STAGE:2022-11-23 01:28:57 6488:6488 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6125935Z STAGE:2022-11-23 01:28:57 6488:6488 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6126268Z STAGE:2022-11-23 01:28:57 6489:6489 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6126590Z STAGE:2022-11-23 01:28:57 6489:6489 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6126698Z ok (6.317s)
2022-11-23T01:48:47.6126718Z 
2022-11-23T01:48:47.6126975Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6127088Z Ran 1 test in 6.317s
2022-11-23T01:48:47.6127108Z 
2022-11-23T01:48:47.6127200Z OK
2022-11-23T01:48:47.6127219Z 
2022-11-23T01:48:47.6127341Z Generating XML reports...
2022-11-23T01:48:47.6127788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012851.xml
2022-11-23T01:48:47.6128156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6128330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6128693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6128884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6128903Z 
2022-11-23T01:48:47.6129101Z Running tests...
2022-11-23T01:48:47.6129373Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6129686Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6129958Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6130174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6606
2022-11-23T01:48:47.6130385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6607
2022-11-23T01:48:47.6130737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6130917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6131292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6131485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6131846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6132016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6132386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6132574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6132816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6133095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6133495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6133891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6134118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6134341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6134671Z STAGE:2022-11-23 01:29:06 6606:6606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6134998Z STAGE:2022-11-23 01:29:06 6607:6607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6135330Z STAGE:2022-11-23 01:29:06 6606:6606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6135666Z STAGE:2022-11-23 01:29:06 6607:6607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6135991Z STAGE:2022-11-23 01:29:06 6606:6606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6136335Z STAGE:2022-11-23 01:29:06 6607:6607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6136655Z STAGE:2022-11-23 01:29:06 6606:6606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6136974Z STAGE:2022-11-23 01:29:06 6607:6607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6137301Z STAGE:2022-11-23 01:29:06 6606:6606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6137640Z STAGE:2022-11-23 01:29:06 6606:6606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6138192Z STAGE:2022-11-23 01:29:06 6607:6607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6138551Z STAGE:2022-11-23 01:29:06 6607:6607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6138868Z STAGE:2022-11-23 01:29:06 6606:6606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6139244Z STAGE:2022-11-23 01:29:06 6607:6607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6139586Z STAGE:2022-11-23 01:29:06 6606:6606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6139907Z STAGE:2022-11-23 01:29:06 6607:6607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6140247Z STAGE:2022-11-23 01:29:06 6606:6606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6140586Z STAGE:2022-11-23 01:29:06 6607:6607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6140689Z ok (6.363s)
2022-11-23T01:48:47.6140709Z 
2022-11-23T01:48:47.6140976Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6141088Z Ran 1 test in 6.364s
2022-11-23T01:48:47.6141107Z 
2022-11-23T01:48:47.6141180Z OK
2022-11-23T01:48:47.6141200Z 
2022-11-23T01:48:47.6141323Z Generating XML reports...
2022-11-23T01:48:47.6141780Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012900.xml
2022-11-23T01:48:47.6142154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6142329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6142709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6142896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6142915Z 
2022-11-23T01:48:47.6143025Z Running tests...
2022-11-23T01:48:47.6143357Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6143652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6143899Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:48:47.6143919Z 
2022-11-23T01:48:47.6144178Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6144292Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6144311Z 
2022-11-23T01:48:47.6144416Z OK (skipped=1)
2022-11-23T01:48:47.6144435Z 
2022-11-23T01:48:47.6144557Z Generating XML reports...
2022-11-23T01:48:47.6144999Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012909.xml
2022-11-23T01:48:47.6145365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6145523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6145902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6146088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6146111Z 
2022-11-23T01:48:47.6146218Z Running tests...
2022-11-23T01:48:47.6146478Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6146791Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6147046Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:48:47.6147065Z 
2022-11-23T01:48:47.6147324Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6147437Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6147456Z 
2022-11-23T01:48:47.6147546Z OK (skipped=1)
2022-11-23T01:48:47.6147583Z 
2022-11-23T01:48:47.6147691Z Generating XML reports...
2022-11-23T01:48:47.6148136Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012912.xml
2022-11-23T01:48:47.6148559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6148739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6149117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6149306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6149326Z 
2022-11-23T01:48:47.6149437Z Running tests...
2022-11-23T01:48:47.6149701Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6149998Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6150248Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6150468Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6790
2022-11-23T01:48:47.6150684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6791
2022-11-23T01:48:47.6151053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6151226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6151605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6151791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6152130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6152301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6152727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6152914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6153159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6153400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6153797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6154193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6154421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6154628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6154734Z ok (6.965s)
2022-11-23T01:48:47.6154754Z 
2022-11-23T01:48:47.6155019Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6155132Z Ran 1 test in 6.965s
2022-11-23T01:48:47.6155151Z 
2022-11-23T01:48:47.6155245Z OK
2022-11-23T01:48:47.6155265Z 
2022-11-23T01:48:47.6155390Z Generating XML reports...
2022-11-23T01:48:47.6155839Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012914.xml
2022-11-23T01:48:47.6156214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6156373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6156751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6156939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6156962Z 
2022-11-23T01:48:47.6157070Z Running tests...
2022-11-23T01:48:47.6157331Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6157705Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6157978Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6158194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6907
2022-11-23T01:48:47.6158404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6908
2022-11-23T01:48:47.6158759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6158936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6159309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6159500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6159861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6160042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6160412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6160596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6160819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6161060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6161458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6161913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6162139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6162365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6163251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py:853: UserWarning: Specified kernel cache directory could not be created! This disables kernel caching. Specified directory is /var/lib/jenkins/.cache/torch/kernels. This warning will appear only once per process. (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cuda/jit_utils.cpp:1444.)
2022-11-23T01:48:47.6163466Z   matches = torch.isclose(actual, expected, rtol=rtol, atol=atol, equal_nan=equal_nan)
2022-11-23T01:48:47.6163568Z ok (7.070s)
2022-11-23T01:48:47.6163590Z 
2022-11-23T01:48:47.6163860Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6163956Z Ran 1 test in 7.070s
2022-11-23T01:48:47.6163994Z 
2022-11-23T01:48:47.6164069Z OK
2022-11-23T01:48:47.6164088Z 
2022-11-23T01:48:47.6164211Z Generating XML reports...
2022-11-23T01:48:47.6164661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012924.xml
2022-11-23T01:48:47.6165028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6165203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6165580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6165768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6165788Z 
2022-11-23T01:48:47.6165895Z Running tests...
2022-11-23T01:48:47.6166145Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6166458Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6166758Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:48:47.6166780Z 
2022-11-23T01:48:47.6167049Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6167161Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6167180Z 
2022-11-23T01:48:47.6167287Z OK (skipped=1)
2022-11-23T01:48:47.6167306Z 
2022-11-23T01:48:47.6167431Z Generating XML reports...
2022-11-23T01:48:47.6167877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012933.xml
2022-11-23T01:48:47.6168248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6168404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6168786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6168974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6168997Z 
2022-11-23T01:48:47.6169105Z Running tests...
2022-11-23T01:48:47.6169367Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6169683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6169949Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6170164Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7057
2022-11-23T01:48:47.6170361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7058
2022-11-23T01:48:47.6170736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6170965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6171342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6171533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6171897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6172069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6172443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6172629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6172854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6173098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6173502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6173905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6174132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6174370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6174592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6174825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6175222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6175602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6175706Z ok (6.966s)
2022-11-23T01:48:47.6175727Z 
2022-11-23T01:48:47.6176037Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6176156Z Ran 1 test in 6.966s
2022-11-23T01:48:47.6176176Z 
2022-11-23T01:48:47.6176267Z OK
2022-11-23T01:48:47.6176286Z 
2022-11-23T01:48:47.6176411Z Generating XML reports...
2022-11-23T01:48:47.6176860Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012936.xml
2022-11-23T01:48:47.6177229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6177385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6177766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6178180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6178202Z 
2022-11-23T01:48:47.6178316Z Running tests...
2022-11-23T01:48:47.6178590Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6178902Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6179152Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:48:47.6179172Z 
2022-11-23T01:48:47.6179433Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6179546Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6179565Z 
2022-11-23T01:48:47.6179656Z OK (skipped=1)
2022-11-23T01:48:47.6179675Z 
2022-11-23T01:48:47.6179799Z Generating XML reports...
2022-11-23T01:48:47.6180250Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012945.xml
2022-11-23T01:48:47.6180711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6180887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6181263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6181452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6181471Z 
2022-11-23T01:48:47.6181579Z Running tests...
2022-11-23T01:48:47.6181841Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6182138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6182398Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6182617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7209
2022-11-23T01:48:47.6182829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7210
2022-11-23T01:48:47.6183203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6183377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6183755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6183945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6184289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6184460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6184832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6185021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6185262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6185561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6185970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6186365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6186591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6186797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6186954Z skip: Skipped due to small world size. (4.350s)
2022-11-23T01:48:47.6186978Z 
2022-11-23T01:48:47.6187244Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6187356Z Ran 1 test in 4.351s
2022-11-23T01:48:47.6187375Z 
2022-11-23T01:48:47.6187480Z OK (skipped=1)
2022-11-23T01:48:47.6187503Z 
2022-11-23T01:48:47.6187627Z Generating XML reports...
2022-11-23T01:48:47.6188079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012948.xml
2022-11-23T01:48:47.6188447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6188622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6188984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6189172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6189240Z 
2022-11-23T01:48:47.6189354Z Running tests...
2022-11-23T01:48:47.6189615Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6189928Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6190213Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:47.6190233Z 
2022-11-23T01:48:47.6190492Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6190602Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6190621Z 
2022-11-23T01:48:47.6190711Z OK (skipped=1)
2022-11-23T01:48:47.6190747Z 
2022-11-23T01:48:47.6190851Z Generating XML reports...
2022-11-23T01:48:47.6191294Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012955.xml
2022-11-23T01:48:47.6191661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6191836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6192212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6192402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6192422Z 
2022-11-23T01:48:47.6192529Z Running tests...
2022-11-23T01:48:47.6192789Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6193082Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6193376Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:47.6193396Z 
2022-11-23T01:48:47.6193655Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6193770Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6193790Z 
2022-11-23T01:48:47.6193897Z OK (skipped=1)
2022-11-23T01:48:47.6193915Z 
2022-11-23T01:48:47.6194038Z Generating XML reports...
2022-11-23T01:48:47.6194536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012957.xml
2022-11-23T01:48:47.6194915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6195090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6195450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6195638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6195658Z 
2022-11-23T01:48:47.6195765Z Running tests...
2022-11-23T01:48:47.6196024Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6196341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6196618Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6197373Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.618s)
2022-11-23T01:48:47.6197394Z 
2022-11-23T01:48:47.6197655Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6197765Z Ran 1 test in 1.618s
2022-11-23T01:48:47.6197784Z 
2022-11-23T01:48:47.6197874Z OK (skipped=1)
2022-11-23T01:48:47.6197912Z 
2022-11-23T01:48:47.6198018Z Generating XML reports...
2022-11-23T01:48:47.6198465Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012959.xml
2022-11-23T01:48:47.6198889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6199063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6199443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6199633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6199688Z 
2022-11-23T01:48:47.6199797Z Running tests...
2022-11-23T01:48:47.6200065Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6200359Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6200647Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6200869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7414
2022-11-23T01:48:47.6201082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7415
2022-11-23T01:48:47.6201453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6201625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6202002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6202190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6202549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6202704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6203073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6203264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6203503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6203792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6204200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6204595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6204822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6205029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6205361Z STAGE:2022-11-23 01:30:09 7415:7415 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6205692Z STAGE:2022-11-23 01:30:09 7414:7414 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6206024Z STAGE:2022-11-23 01:30:10 7414:7414 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6206353Z STAGE:2022-11-23 01:30:10 7415:7415 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6206693Z STAGE:2022-11-23 01:30:10 7414:7414 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6207037Z STAGE:2022-11-23 01:30:10 7415:7415 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6207141Z ok (7.710s)
2022-11-23T01:48:47.6207161Z 
2022-11-23T01:48:47.6207422Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6207516Z Ran 1 test in 7.711s
2022-11-23T01:48:47.6207536Z 
2022-11-23T01:48:47.6207627Z OK
2022-11-23T01:48:47.6207697Z 
2022-11-23T01:48:47.6207830Z Generating XML reports...
2022-11-23T01:48:47.6208280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013003.xml
2022-11-23T01:48:47.6208650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6208823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6209198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6209388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6209407Z 
2022-11-23T01:48:47.6209515Z Running tests...
2022-11-23T01:48:47.6209761Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6210071Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6210371Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:47.6210391Z 
2022-11-23T01:48:47.6210650Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6210765Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6210784Z 
2022-11-23T01:48:47.6210890Z OK (skipped=1)
2022-11-23T01:48:47.6210908Z 
2022-11-23T01:48:47.6211031Z Generating XML reports...
2022-11-23T01:48:47.6211476Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013014.xml
2022-11-23T01:48:47.6211844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6212002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6212380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6212572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6212592Z 
2022-11-23T01:48:47.6212699Z Running tests...
2022-11-23T01:48:47.6212958Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6213324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6213624Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6214366Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-11-23T01:48:47.6214387Z 
2022-11-23T01:48:47.6214644Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6214742Z Ran 1 test in 1.613s
2022-11-23T01:48:47.6214779Z 
2022-11-23T01:48:47.6214868Z OK (skipped=1)
2022-11-23T01:48:47.6214887Z 
2022-11-23T01:48:47.6215009Z Generating XML reports...
2022-11-23T01:48:47.6215455Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013016.xml
2022-11-23T01:48:47.6215824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6250438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6250946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6251134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6251157Z 
2022-11-23T01:48:47.6251258Z Running tests...
2022-11-23T01:48:47.6251513Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6252007Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6252288Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:47.6252315Z 
2022-11-23T01:48:47.6252566Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6252669Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6252689Z 
2022-11-23T01:48:47.6252785Z OK (skipped=1)
2022-11-23T01:48:47.6252804Z 
2022-11-23T01:48:47.6252916Z Generating XML reports...
2022-11-23T01:48:47.6253356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013020.xml
2022-11-23T01:48:47.6253721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6253877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6254249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6254426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6254445Z 
2022-11-23T01:48:47.6254545Z Running tests...
2022-11-23T01:48:47.6254800Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6255107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6255392Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6255603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7635
2022-11-23T01:48:47.6255801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7636
2022-11-23T01:48:47.6256170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6256349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6256719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6256990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6257371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6257548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6258157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6258363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6258589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6258832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6259227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6259615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6259832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6260047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6260194Z skip: Skipped due to small world size. (4.255s)
2022-11-23T01:48:47.6260214Z 
2022-11-23T01:48:47.6260467Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6260568Z Ran 1 test in 4.255s
2022-11-23T01:48:47.6260588Z 
2022-11-23T01:48:47.6260679Z OK (skipped=1)
2022-11-23T01:48:47.6260697Z 
2022-11-23T01:48:47.6260904Z Generating XML reports...
2022-11-23T01:48:47.6261352Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013023.xml
2022-11-23T01:48:47.6261716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6261879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6262245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6262424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6262444Z 
2022-11-23T01:48:47.6262543Z Running tests...
2022-11-23T01:48:47.6262786Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6263091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6263363Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:47.6263389Z 
2022-11-23T01:48:47.6263637Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6263738Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6263761Z 
2022-11-23T01:48:47.6263858Z OK (skipped=1)
2022-11-23T01:48:47.6263878Z 
2022-11-23T01:48:47.6263990Z Generating XML reports...
2022-11-23T01:48:47.6264426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013029.xml
2022-11-23T01:48:47.6264782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6264938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6265302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6265484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6265504Z 
2022-11-23T01:48:47.6265605Z Running tests...
2022-11-23T01:48:47.6265857Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6266230Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6266537Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:47.6266558Z 
2022-11-23T01:48:47.6266809Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6266915Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6266934Z 
2022-11-23T01:48:47.6267024Z OK (skipped=1)
2022-11-23T01:48:47.6267062Z 
2022-11-23T01:48:47.6267167Z Generating XML reports...
2022-11-23T01:48:47.6267604Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013032.xml
2022-11-23T01:48:47.6267968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6268141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6268513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6268695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6268715Z 
2022-11-23T01:48:47.6268825Z Running tests...
2022-11-23T01:48:47.6269075Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6269369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6269639Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6269851Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7806
2022-11-23T01:48:47.6270122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7807
2022-11-23T01:48:47.6270484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6270655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6271029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6271213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6271558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6271731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6272098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6272285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6272526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6272762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6273152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6273549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6273771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6273977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6274073Z ok (6.870s)
2022-11-23T01:48:47.6274093Z 
2022-11-23T01:48:47.6274348Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6274458Z Ran 1 test in 6.870s
2022-11-23T01:48:47.6274478Z 
2022-11-23T01:48:47.6274572Z OK
2022-11-23T01:48:47.6274591Z 
2022-11-23T01:48:47.6274707Z Generating XML reports...
2022-11-23T01:48:47.6275194Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013034.xml
2022-11-23T01:48:47.6275574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6275778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6276135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6276315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6276334Z 
2022-11-23T01:48:47.6276435Z Running tests...
2022-11-23T01:48:47.6276688Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6277004Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6277291Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6277501Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7923
2022-11-23T01:48:47.6277706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7924
2022-11-23T01:48:47.6278055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6278221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6278593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6278773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6279189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6279359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6279729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6279910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6280151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6280372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6280762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6281147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6281379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6281594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6281690Z ok (7.030s)
2022-11-23T01:48:47.6281713Z 
2022-11-23T01:48:47.6281978Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6282084Z Ran 1 test in 7.030s
2022-11-23T01:48:47.6282104Z 
2022-11-23T01:48:47.6282178Z OK
2022-11-23T01:48:47.6282207Z 
2022-11-23T01:48:47.6282314Z Generating XML reports...
2022-11-23T01:48:47.6282759Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml
2022-11-23T01:48:47.6283118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6283284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6283662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6283843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6283863Z 
2022-11-23T01:48:47.6284015Z Running tests...
2022-11-23T01:48:47.6284288Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6284581Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6284873Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:47.6284893Z 
2022-11-23T01:48:47.6285143Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6285254Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6285273Z 
2022-11-23T01:48:47.6285380Z OK (skipped=1)
2022-11-23T01:48:47.6285399Z 
2022-11-23T01:48:47.6285519Z Generating XML reports...
2022-11-23T01:48:47.6285956Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013053.xml
2022-11-23T01:48:47.6286330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6286499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6286855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6287039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6287059Z 
2022-11-23T01:48:47.6287168Z Running tests...
2022-11-23T01:48:47.6287419Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6287725Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6288073Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6288286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8073
2022-11-23T01:48:47.6288495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8074
2022-11-23T01:48:47.6288849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6289026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6289390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6289570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6289933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6290096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6290471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6290657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6290894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6291117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6291509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6291902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6292121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6292349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6292575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6292848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6293243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6293629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6293715Z ok (6.900s)
2022-11-23T01:48:47.6293735Z 
2022-11-23T01:48:47.6293996Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6294102Z Ran 1 test in 6.900s
2022-11-23T01:48:47.6294121Z 
2022-11-23T01:48:47.6294213Z OK
2022-11-23T01:48:47.6294232Z 
2022-11-23T01:48:47.6294349Z Generating XML reports...
2022-11-23T01:48:47.6294792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013056.xml
2022-11-23T01:48:47.6295154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6295325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6295703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6295874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6295893Z 
2022-11-23T01:48:47.6296002Z Running tests...
2022-11-23T01:48:47.6296257Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6296561Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6296852Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:47.6296924Z 
2022-11-23T01:48:47.6297190Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6297306Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6297325Z 
2022-11-23T01:48:47.6297438Z OK (skipped=1)
2022-11-23T01:48:47.6297458Z 
2022-11-23T01:48:47.6297563Z Generating XML reports...
2022-11-23T01:48:47.6298253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013105.xml
2022-11-23T01:48:47.6298636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6298811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6299183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6299372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6299398Z 
2022-11-23T01:48:47.6299509Z Running tests...
2022-11-23T01:48:47.6299801Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6300120Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6300387Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6300606Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8225
2022-11-23T01:48:47.6300818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8226
2022-11-23T01:48:47.6301181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6301354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6301728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6301921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6302286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6302522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6302912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6303101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6303346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6303588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6303981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6304379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6304604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6304832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6304972Z skip: Skipped due to small world size. (4.258s)
2022-11-23T01:48:47.6305008Z 
2022-11-23T01:48:47.6305256Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6305367Z Ran 1 test in 4.258s
2022-11-23T01:48:47.6305387Z 
2022-11-23T01:48:47.6305494Z OK (skipped=1)
2022-11-23T01:48:47.6305513Z 
2022-11-23T01:48:47.6305637Z Generating XML reports...
2022-11-23T01:48:47.6306085Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013107.xml
2022-11-23T01:48:47.6306566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6306740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6307118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6307288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6307309Z 
2022-11-23T01:48:47.6307419Z Running tests...
2022-11-23T01:48:47.6307682Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6307993Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6308256Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6308473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8330
2022-11-23T01:48:47.6308691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8331
2022-11-23T01:48:47.6309119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6309278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6309649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6309838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6310201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6310370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6310746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6310932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6311180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6311422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6311853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6312260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6312486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6312709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6312949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6313184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6313581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6313971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6314076Z ok (6.966s)
2022-11-23T01:48:47.6314097Z 
2022-11-23T01:48:47.6314344Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6314457Z Ran 1 test in 6.966s
2022-11-23T01:48:47.6314477Z 
2022-11-23T01:48:47.6314569Z OK
2022-11-23T01:48:47.6314588Z 
2022-11-23T01:48:47.6314711Z Generating XML reports...
2022-11-23T01:48:47.6315159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013114.xml
2022-11-23T01:48:47.6315526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6315761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6316146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6316339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6316360Z 
2022-11-23T01:48:47.6316452Z Running tests...
2022-11-23T01:48:47.6316719Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6317030Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6317290Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6317506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8454
2022-11-23T01:48:47.6317720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8455
2022-11-23T01:48:47.6318094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6318270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6318630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6318818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6319182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6319352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6319725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6319914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6320154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6320400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6320868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6321252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6321479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6321703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6321852Z skip: Need at least 3 CUDA devices (4.315s)
2022-11-23T01:48:47.6321873Z 
2022-11-23T01:48:47.6322131Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6322245Z Ran 1 test in 4.315s
2022-11-23T01:48:47.6322269Z 
2022-11-23T01:48:47.6322379Z OK (skipped=1)
2022-11-23T01:48:47.6322398Z 
2022-11-23T01:48:47.6322523Z Generating XML reports...
2022-11-23T01:48:47.6322952Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013124.xml
2022-11-23T01:48:47.6323323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6323499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6323871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6324058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6324077Z 
2022-11-23T01:48:47.6324185Z Running tests...
2022-11-23T01:48:47.6324445Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6324758Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6325068Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-11-23T01:48:47.6325089Z 
2022-11-23T01:48:47.6325338Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6325453Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6325472Z 
2022-11-23T01:48:47.6325581Z OK (skipped=1)
2022-11-23T01:48:47.6325600Z 
2022-11-23T01:48:47.6325725Z Generating XML reports...
2022-11-23T01:48:47.6326172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013131.xml
2022-11-23T01:48:47.6326542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6326716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6327090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6327283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6327302Z 
2022-11-23T01:48:47.6327393Z Running tests...
2022-11-23T01:48:47.6327657Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6327968Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6328219Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T01:48:47.6328238Z 
2022-11-23T01:48:47.6328496Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6328607Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6328626Z 
2022-11-23T01:48:47.6328732Z OK (skipped=1)
2022-11-23T01:48:47.6328751Z 
2022-11-23T01:48:47.6328873Z Generating XML reports...
2022-11-23T01:48:47.6329297Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013133.xml
2022-11-23T01:48:47.6329670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6329843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6330268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6330463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6330483Z 
2022-11-23T01:48:47.6330595Z Running tests...
2022-11-23T01:48:47.6330855Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6331167Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6331417Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6331615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8625
2022-11-23T01:48:47.6331834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8626
2022-11-23T01:48:47.6332202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6332378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6332752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6332941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6333303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6333474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6333828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6334074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6334315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6334563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6334976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6335371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6335599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6335823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6335928Z ok (7.614s)
2022-11-23T01:48:47.6335949Z 
2022-11-23T01:48:47.6336192Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6336310Z Ran 1 test in 7.614s
2022-11-23T01:48:47.6336329Z 
2022-11-23T01:48:47.6336420Z OK
2022-11-23T01:48:47.6336440Z 
2022-11-23T01:48:47.6336563Z Generating XML reports...
2022-11-23T01:48:47.6337011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013135.xml
2022-11-23T01:48:47.6337379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6337551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6338157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6338357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6338379Z 
2022-11-23T01:48:47.6338470Z Running tests...
2022-11-23T01:48:47.6338739Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6339061Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6339325Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T01:48:47.6339430Z 
2022-11-23T01:48:47.6339704Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6339819Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6339838Z 
2022-11-23T01:48:47.6339948Z OK (skipped=1)
2022-11-23T01:48:47.6339967Z 
2022-11-23T01:48:47.6340091Z Generating XML reports...
2022-11-23T01:48:47.6340520Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013146.xml
2022-11-23T01:48:47.6340891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6341065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6341446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6341633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6341652Z 
2022-11-23T01:48:47.6341763Z Running tests...
2022-11-23T01:48:47.6342024Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6342334Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6342599Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6342796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8773
2022-11-23T01:48:47.6343011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8774
2022-11-23T01:48:47.6343378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6343623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6344004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6344197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6344563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6344736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6345092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6345278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6345517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6345761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6346157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6346552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6346779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6347002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6347158Z skip: Skipped due to small world size. (4.251s)
2022-11-23T01:48:47.6347178Z 
2022-11-23T01:48:47.6347425Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6347538Z Ran 1 test in 4.251s
2022-11-23T01:48:47.6347557Z 
2022-11-23T01:48:47.6347665Z OK (skipped=1)
2022-11-23T01:48:47.6347689Z 
2022-11-23T01:48:47.6347811Z Generating XML reports...
2022-11-23T01:48:47.6348259Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013148.xml
2022-11-23T01:48:47.6348674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6348855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6349237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6349427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6349447Z 
2022-11-23T01:48:47.6349538Z Running tests...
2022-11-23T01:48:47.6349798Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6350109Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6350369Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T01:48:47.6350388Z 
2022-11-23T01:48:47.6350643Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6350753Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6350773Z 
2022-11-23T01:48:47.6350883Z OK (skipped=1)
2022-11-23T01:48:47.6350902Z 
2022-11-23T01:48:47.6351026Z Generating XML reports...
2022-11-23T01:48:47.6351468Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013155.xml
2022-11-23T01:48:47.6351815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6351988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6352365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6352606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6352626Z 
2022-11-23T01:48:47.6352736Z Running tests...
2022-11-23T01:48:47.6352999Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6353317Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6353576Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6353774Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8911
2022-11-23T01:48:47.6353992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8912
2022-11-23T01:48:47.6354354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6354527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6354897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6355088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6355461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6355635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6356007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6356176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6356416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6356657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6357055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6357455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6357731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6357964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6358125Z skip: Skipped due to small world size. (4.257s)
2022-11-23T01:48:47.6358145Z 
2022-11-23T01:48:47.6358412Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6358507Z Ran 1 test in 4.257s
2022-11-23T01:48:47.6358526Z 
2022-11-23T01:48:47.6358636Z OK (skipped=1)
2022-11-23T01:48:47.6358655Z 
2022-11-23T01:48:47.6358779Z Generating XML reports...
2022-11-23T01:48:47.6359226Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013157.xml
2022-11-23T01:48:47.6359599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6359772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6360159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6360346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6360366Z 
2022-11-23T01:48:47.6360457Z Running tests...
2022-11-23T01:48:47.6360715Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6361027Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6361301Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T01:48:47.6361321Z 
2022-11-23T01:48:47.6361579Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6361746Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6361765Z 
2022-11-23T01:48:47.6361875Z OK (skipped=1)
2022-11-23T01:48:47.6361894Z 
2022-11-23T01:48:47.6362020Z Generating XML reports...
2022-11-23T01:48:47.6362465Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013204.xml
2022-11-23T01:48:47.6362813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6362986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6363358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6363544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6363563Z 
2022-11-23T01:48:47.6363671Z Running tests...
2022-11-23T01:48:47.6363926Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6364245Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6364516Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T01:48:47.6364539Z 
2022-11-23T01:48:47.6364798Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6364893Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6364913Z 
2022-11-23T01:48:47.6365022Z OK (skipped=1)
2022-11-23T01:48:47.6365040Z 
2022-11-23T01:48:47.6365164Z Generating XML reports...
2022-11-23T01:48:47.6365607Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013206.xml
2022-11-23T01:48:47.6365971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6366143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6366524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6366712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6366732Z 
2022-11-23T01:48:47.6366889Z Running tests...
2022-11-23T01:48:47.6367141Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6367450Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6367720Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T01:48:47.6367740Z 
2022-11-23T01:48:47.6367997Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6368112Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6368131Z 
2022-11-23T01:48:47.6368241Z OK (skipped=1)
2022-11-23T01:48:47.6368260Z 
2022-11-23T01:48:47.6368388Z Generating XML reports...
2022-11-23T01:48:47.6368832Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013209.xml
2022-11-23T01:48:47.6369182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6369356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6369732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6369921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6369940Z 
2022-11-23T01:48:47.6370048Z Running tests...
2022-11-23T01:48:47.6370309Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6370618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6370870Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-11-23T01:48:47.6370939Z 
2022-11-23T01:48:47.6371204Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6371300Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6371319Z 
2022-11-23T01:48:47.6371432Z OK (skipped=1)
2022-11-23T01:48:47.6371452Z 
2022-11-23T01:48:47.6371576Z Generating XML reports...
2022-11-23T01:48:47.6372018Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013211.xml
2022-11-23T01:48:47.6372380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6372556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6372931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6373120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6373145Z 
2022-11-23T01:48:47.6373254Z Running tests...
2022-11-23T01:48:47.6373494Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6373804Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6374065Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-11-23T01:48:47.6374085Z 
2022-11-23T01:48:47.6374342Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6374455Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6374474Z 
2022-11-23T01:48:47.6374586Z OK (skipped=1)
2022-11-23T01:48:47.6374606Z 
2022-11-23T01:48:47.6374729Z Generating XML reports...
2022-11-23T01:48:47.6375170Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013214.xml
2022-11-23T01:48:47.6375520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6375699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6376122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6376317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6376337Z 
2022-11-23T01:48:47.6376447Z Running tests...
2022-11-23T01:48:47.6376708Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6377020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6377295Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6377518Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9181
2022-11-23T01:48:47.6377718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9182
2022-11-23T01:48:47.6378315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6378505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6378884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6379074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6379435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6379609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6379980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6380149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6380478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6380722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6381129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6381524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6381754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6381978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6382219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6382457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6382836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6383228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6383468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:47.6383704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:47.6384089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.6384475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.6384579Z ok (4.488s)
2022-11-23T01:48:47.6384599Z 
2022-11-23T01:48:47.6384865Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6384978Z Ran 1 test in 4.488s
2022-11-23T01:48:47.6384997Z 
2022-11-23T01:48:47.6385072Z OK
2022-11-23T01:48:47.6385108Z 
2022-11-23T01:48:47.6385213Z Generating XML reports...
2022-11-23T01:48:47.6385735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013216.xml
2022-11-23T01:48:47.6386117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6386293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6386670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6386860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6386880Z 
2022-11-23T01:48:47.6386990Z Running tests...
2022-11-23T01:48:47.6387266Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6387563Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6387828Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6388045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9294
2022-11-23T01:48:47.6388257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9295
2022-11-23T01:48:47.6388625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6388796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6389169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6389355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6389763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6389938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6390313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6390499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6390742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6390983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6391378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6391768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6391998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6392205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6392311Z ok (5.951s)
2022-11-23T01:48:47.6392331Z 
2022-11-23T01:48:47.6392597Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6392709Z Ran 1 test in 5.951s
2022-11-23T01:48:47.6392728Z 
2022-11-23T01:48:47.6392818Z OK
2022-11-23T01:48:47.6392837Z 
2022-11-23T01:48:47.6392960Z Generating XML reports...
2022-11-23T01:48:47.6393405Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013223.xml
2022-11-23T01:48:47.6393770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6393942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6394307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6394495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6394515Z 
2022-11-23T01:48:47.6394672Z Running tests...
2022-11-23T01:48:47.6394941Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6395255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6395531Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6395750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9410
2022-11-23T01:48:47.6395965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9411
2022-11-23T01:48:47.6396316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6396496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6396873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6397062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6397426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6397600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6397976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6398165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6398406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6398684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6399083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6399478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6399743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6399970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6400130Z skip: Skipped due to small world size. (4.256s)
2022-11-23T01:48:47.6400150Z 
2022-11-23T01:48:47.6400418Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6400532Z Ran 1 test in 4.256s
2022-11-23T01:48:47.6400551Z 
2022-11-23T01:48:47.6400641Z OK (skipped=1)
2022-11-23T01:48:47.6400684Z 
2022-11-23T01:48:47.6400792Z Generating XML reports...
2022-11-23T01:48:47.6401237Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013231.xml
2022-11-23T01:48:47.6401610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6401783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6402149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6402336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6402355Z 
2022-11-23T01:48:47.6402464Z Running tests...
2022-11-23T01:48:47.6402721Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6403016Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6403285Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6403501Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9515
2022-11-23T01:48:47.6403765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9516
2022-11-23T01:48:47.6404146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6404321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6404698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6404887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6405250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6405404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6405781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6405968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6406211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6406451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6406847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6407241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6407466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6407677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6407834Z ok (5.664s)
2022-11-23T01:48:47.6407854Z 
2022-11-23T01:48:47.6408118Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6408232Z Ran 1 test in 5.665s
2022-11-23T01:48:47.6408255Z 
2022-11-23T01:48:47.6408350Z OK
2022-11-23T01:48:47.6408369Z 
2022-11-23T01:48:47.6408495Z Generating XML reports...
2022-11-23T01:48:47.6408941Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013238.xml
2022-11-23T01:48:47.6409312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6409487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6409842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6410031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6410055Z 
2022-11-23T01:48:47.6410162Z Running tests...
2022-11-23T01:48:47.6410423Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6410739Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6411010Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6411224Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9621
2022-11-23T01:48:47.6411436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9622
2022-11-23T01:48:47.6411785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6411959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6412335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6412530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6412948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6413129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6413506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6413694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6413934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6414156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6414554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6414954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6415181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6415406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6415508Z ok (4.345s)
2022-11-23T01:48:47.6415528Z 
2022-11-23T01:48:47.6415790Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6415903Z Ran 1 test in 4.346s
2022-11-23T01:48:47.6415923Z 
2022-11-23T01:48:47.6415998Z OK
2022-11-23T01:48:47.6416034Z 
2022-11-23T01:48:47.6416141Z Generating XML reports...
2022-11-23T01:48:47.6416588Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013246.xml
2022-11-23T01:48:47.6416956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6417185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6417564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6417753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6417773Z 
2022-11-23T01:48:47.6418127Z Running tests...
2022-11-23T01:48:47.6418413Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6418710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6418998Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6419216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9726
2022-11-23T01:48:47.6419435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9727
2022-11-23T01:48:47.6419805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6419977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6420359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6420549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6420918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6421073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6421448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6421633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6421879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6422119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6422592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6423003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6423230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6423436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6423538Z ok (5.920s)
2022-11-23T01:48:47.6423560Z 
2022-11-23T01:48:47.6423824Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6423943Z Ran 1 test in 5.920s
2022-11-23T01:48:47.6423962Z 
2022-11-23T01:48:47.6424053Z OK
2022-11-23T01:48:47.6424072Z 
2022-11-23T01:48:47.6424194Z Generating XML reports...
2022-11-23T01:48:47.6424644Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013253.xml
2022-11-23T01:48:47.6425014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6425187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6425544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6425731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6425751Z 
2022-11-23T01:48:47.6425859Z Running tests...
2022-11-23T01:48:47.6426123Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6426514Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6426781Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6427001Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9842
2022-11-23T01:48:47.6427218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9843
2022-11-23T01:48:47.6427566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6427740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6428114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6428300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6428663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6428842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6429213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6429402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6429641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6429864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6430259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6430650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6430875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6431104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6431206Z ok (6.044s)
2022-11-23T01:48:47.6431226Z 
2022-11-23T01:48:47.6431541Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6431665Z Ran 1 test in 6.044s
2022-11-23T01:48:47.6431685Z 
2022-11-23T01:48:47.6431759Z OK
2022-11-23T01:48:47.6431800Z 
2022-11-23T01:48:47.6431906Z Generating XML reports...
2022-11-23T01:48:47.6432353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013302.xml
2022-11-23T01:48:47.6432721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6432895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6433269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6433464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6433483Z 
2022-11-23T01:48:47.6433592Z Running tests...
2022-11-23T01:48:47.6433858Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6434151Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6434421Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6434635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9956
2022-11-23T01:48:47.6434848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9957
2022-11-23T01:48:47.6435215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6435390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6435826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6436015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6436383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6436538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6436909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6437096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6437336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6437579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6437979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6438377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6438606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6438810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6438914Z ok (9.526s)
2022-11-23T01:48:47.6438933Z 
2022-11-23T01:48:47.6439198Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6439311Z Ran 1 test in 9.527s
2022-11-23T01:48:47.6439331Z 
2022-11-23T01:48:47.6439423Z OK
2022-11-23T01:48:47.6439442Z 
2022-11-23T01:48:47.6439564Z Generating XML reports...
2022-11-23T01:48:47.6440010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013315.xml
2022-11-23T01:48:47.6440380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6440554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6440957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6441156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6441176Z 
2022-11-23T01:48:47.6441287Z Running tests...
2022-11-23T01:48:47.6441550Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6441862Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6442115Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6442135Z 
2022-11-23T01:48:47.6442398Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6442510Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6442529Z 
2022-11-23T01:48:47.6442621Z OK (skipped=1)
2022-11-23T01:48:47.6442660Z 
2022-11-23T01:48:47.6442765Z Generating XML reports...
2022-11-23T01:48:47.6443217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013327.xml
2022-11-23T01:48:47.6443584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6443755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6444131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6444319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6444339Z 
2022-11-23T01:48:47.6444449Z Running tests...
2022-11-23T01:48:47.6444769Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6445063Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6445318Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6446068Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.656s)
2022-11-23T01:48:47.6446090Z 
2022-11-23T01:48:47.6446353Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6446466Z Ran 1 test in 1.656s
2022-11-23T01:48:47.6446485Z 
2022-11-23T01:48:47.6446595Z OK (skipped=1)
2022-11-23T01:48:47.6446614Z 
2022-11-23T01:48:47.6446739Z Generating XML reports...
2022-11-23T01:48:47.6447186Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013330.xml
2022-11-23T01:48:47.6447554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6447732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6448088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6448277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6448297Z 
2022-11-23T01:48:47.6448405Z Running tests...
2022-11-23T01:48:47.6448663Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6448977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6449241Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6449265Z 
2022-11-23T01:48:47.6449526Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6449637Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6449657Z 
2022-11-23T01:48:47.6449747Z OK (skipped=1)
2022-11-23T01:48:47.6449881Z 
2022-11-23T01:48:47.6449996Z Generating XML reports...
2022-11-23T01:48:47.6450450Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013334.xml
2022-11-23T01:48:47.6450817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6450993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6451370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6451557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6451581Z 
2022-11-23T01:48:47.6451692Z Running tests...
2022-11-23T01:48:47.6451952Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6452246Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6452511Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6452531Z 
2022-11-23T01:48:47.6452788Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6452899Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6452918Z 
2022-11-23T01:48:47.6453026Z OK (skipped=1)
2022-11-23T01:48:47.6453045Z 
2022-11-23T01:48:47.6453166Z Generating XML reports...
2022-11-23T01:48:47.6453613Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013336.xml
2022-11-23T01:48:47.6453977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6454207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6454567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6454759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6454779Z 
2022-11-23T01:48:47.6454887Z Running tests...
2022-11-23T01:48:47.6455147Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6455455Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6455718Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-11-23T01:48:47.6455739Z 
2022-11-23T01:48:47.6456003Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6456116Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6456140Z 
2022-11-23T01:48:47.6456248Z OK (skipped=1)
2022-11-23T01:48:47.6456268Z 
2022-11-23T01:48:47.6456374Z Generating XML reports...
2022-11-23T01:48:47.6456825Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013339.xml
2022-11-23T01:48:47.6457195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6457368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6457741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6458219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6458244Z 
2022-11-23T01:48:47.6458363Z Running tests...
2022-11-23T01:48:47.6458635Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6458930Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6459198Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6460023Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.654s)
2022-11-23T01:48:47.6460046Z 
2022-11-23T01:48:47.6460320Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6460433Z Ran 1 test in 1.654s
2022-11-23T01:48:47.6460453Z 
2022-11-23T01:48:47.6460560Z OK (skipped=1)
2022-11-23T01:48:47.6460579Z 
2022-11-23T01:48:47.6460704Z Generating XML reports...
2022-11-23T01:48:47.6461146Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013341.xml
2022-11-23T01:48:47.6461521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6461695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6462054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6462244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6462264Z 
2022-11-23T01:48:47.6462374Z Running tests...
2022-11-23T01:48:47.6462631Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6462942Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6463251Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6463992Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-11-23T01:48:47.6464079Z 
2022-11-23T01:48:47.6464349Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6464464Z Ran 1 test in 1.640s
2022-11-23T01:48:47.6464483Z 
2022-11-23T01:48:47.6464590Z OK (skipped=1)
2022-11-23T01:48:47.6464610Z 
2022-11-23T01:48:47.6464715Z Generating XML reports...
2022-11-23T01:48:47.6465157Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013345.xml
2022-11-23T01:48:47.6465522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6465693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6466076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6466265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6466284Z 
2022-11-23T01:48:47.6466395Z Running tests...
2022-11-23T01:48:47.6466655Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6466949Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6467260Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6467998Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.611s)
2022-11-23T01:48:47.6468023Z 
2022-11-23T01:48:47.6468285Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6468398Z Ran 1 test in 1.611s
2022-11-23T01:48:47.6468417Z 
2022-11-23T01:48:47.6468526Z OK (skipped=1)
2022-11-23T01:48:47.6468545Z 
2022-11-23T01:48:47.6468719Z Generating XML reports...
2022-11-23T01:48:47.6469171Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013350.xml
2022-11-23T01:48:47.6469540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6469714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6470068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6470257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6470282Z 
2022-11-23T01:48:47.6470392Z Running tests...
2022-11-23T01:48:47.6470651Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6470961Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6471225Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6471444Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10329
2022-11-23T01:48:47.6471664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10330
2022-11-23T01:48:47.6472042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6472199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6472577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6472821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6473189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6473366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6473744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6473935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6474177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6474401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6474798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6475194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6475421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6475646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6475898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj_yx7f0
2022-11-23T01:48:47.6476161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj_yx7f0/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6476411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq7rmvvi
2022-11-23T01:48:47.6476674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq7rmvvi/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6476888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6477127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6477233Z ok (6.262s)
2022-11-23T01:48:47.6477253Z 
2022-11-23T01:48:47.6477521Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6477633Z Ran 1 test in 6.262s
2022-11-23T01:48:47.6477701Z 
2022-11-23T01:48:47.6477800Z OK
2022-11-23T01:48:47.6477819Z 
2022-11-23T01:48:47.6477944Z Generating XML reports...
2022-11-23T01:48:47.6478395Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013354.xml
2022-11-23T01:48:47.6478745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6478921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6479298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6479491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6479510Z 
2022-11-23T01:48:47.6479617Z Running tests...
2022-11-23T01:48:47.6479878Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6480194Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6480465Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6480681Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10447
2022-11-23T01:48:47.6480877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10448
2022-11-23T01:48:47.6481243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6481415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6481792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6482036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6482408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6482585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6482957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6483126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6483367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6483608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6484006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6484403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6484634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6484861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6485115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuyyqidb
2022-11-23T01:48:47.6485382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuyyqidb/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6485613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyeqipi4
2022-11-23T01:48:47.6485875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyeqipi4/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6486104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6486340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6486567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6486848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6486960Z ok (6.359s)
2022-11-23T01:48:47.6486980Z 
2022-11-23T01:48:47.6487249Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6487344Z Ran 1 test in 6.360s
2022-11-23T01:48:47.6487383Z 
2022-11-23T01:48:47.6487457Z OK
2022-11-23T01:48:47.6487476Z 
2022-11-23T01:48:47.6487600Z Generating XML reports...
2022-11-23T01:48:47.6488048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013403.xml
2022-11-23T01:48:47.6488416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6488596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6488971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6489162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6489181Z 
2022-11-23T01:48:47.6489291Z Running tests...
2022-11-23T01:48:47.6489536Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6489847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6490116Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6490862Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.639s)
2022-11-23T01:48:47.6490934Z 
2022-11-23T01:48:47.6491203Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6491322Z Ran 1 test in 1.639s
2022-11-23T01:48:47.6491341Z 
2022-11-23T01:48:47.6491451Z OK (skipped=1)
2022-11-23T01:48:47.6491469Z 
2022-11-23T01:48:47.6491592Z Generating XML reports...
2022-11-23T01:48:47.6492037Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013411.xml
2022-11-23T01:48:47.6492405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6492560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6492932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6493123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6493142Z 
2022-11-23T01:48:47.6493249Z Running tests...
2022-11-23T01:48:47.6493507Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6493820Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6494106Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6494843Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s)
2022-11-23T01:48:47.6494865Z 
2022-11-23T01:48:47.6495122Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6495220Z Ran 1 test in 1.658s
2022-11-23T01:48:47.6495259Z 
2022-11-23T01:48:47.6495348Z OK (skipped=1)
2022-11-23T01:48:47.6495367Z 
2022-11-23T01:48:47.6495492Z Generating XML reports...
2022-11-23T01:48:47.6495979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013416.xml
2022-11-23T01:48:47.6496359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6496532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6496901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6497088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6497108Z 
2022-11-23T01:48:47.6497216Z Running tests...
2022-11-23T01:48:47.6497457Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6497772Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6498286Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6498517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10633
2022-11-23T01:48:47.6498734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10634
2022-11-23T01:48:47.6499109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6499284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6499660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6499880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6500229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6500495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6500875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6501063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6501310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6501554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6501952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6502345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6502559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6502782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6503039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy314bq7
2022-11-23T01:48:47.6503302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy314bq7/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6503552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfu09vs2
2022-11-23T01:48:47.6503815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfu09vs2/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6504025Z <class 'torch.nn.parameter.Parameter'>
2022-11-23T01:48:47.6504129Z ok (7.080s)
2022-11-23T01:48:47.6504150Z 
2022-11-23T01:48:47.6504415Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6504511Z Ran 1 test in 7.081s
2022-11-23T01:48:47.6504536Z 
2022-11-23T01:48:47.6504630Z OK
2022-11-23T01:48:47.6504649Z 
2022-11-23T01:48:47.6504773Z Generating XML reports...
2022-11-23T01:48:47.6505220Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013420.xml
2022-11-23T01:48:47.6505656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6505845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6506228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6506418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6506440Z 
2022-11-23T01:48:47.6506531Z Running tests...
2022-11-23T01:48:47.6506796Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6507108Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6507414Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6507632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10748
2022-11-23T01:48:47.6507847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10749
2022-11-23T01:48:47.6508216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6508390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6508763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6508932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6509296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6509525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6509905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6510097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6510340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6510583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6510982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6511375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6511585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6511815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6512069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbye7u42
2022-11-23T01:48:47.6512339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbye7u42/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6512590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc74deivz
2022-11-23T01:48:47.6512852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc74deivz/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6512954Z ok (6.920s)
2022-11-23T01:48:47.6512974Z 
2022-11-23T01:48:47.6513240Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6513336Z Ran 1 test in 6.920s
2022-11-23T01:48:47.6513356Z 
2022-11-23T01:48:47.6513448Z OK
2022-11-23T01:48:47.6513468Z 
2022-11-23T01:48:47.6513596Z Generating XML reports...
2022-11-23T01:48:47.6514043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013429.xml
2022-11-23T01:48:47.6514477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6514661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6515038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6515229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6515249Z 
2022-11-23T01:48:47.6515359Z Running tests...
2022-11-23T01:48:47.6515602Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6515915Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6516172Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6516393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10863
2022-11-23T01:48:47.6516606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10864
2022-11-23T01:48:47.6516977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6517151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6517528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6517697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6518063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6518235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6518673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6518860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6519106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6519347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6519743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6520135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6520344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6520571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6520830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllmanyuv
2022-11-23T01:48:47.6521099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllmanyuv/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6521351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc3rl50m
2022-11-23T01:48:47.6521615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc3rl50m/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6521847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6522077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6522161Z ok (7.553s)
2022-11-23T01:48:47.6522198Z 
2022-11-23T01:48:47.6522453Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6522567Z Ran 1 test in 7.554s
2022-11-23T01:48:47.6522592Z 
2022-11-23T01:48:47.6522684Z OK
2022-11-23T01:48:47.6522704Z 
2022-11-23T01:48:47.6522828Z Generating XML reports...
2022-11-23T01:48:47.6523275Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013439.xml
2022-11-23T01:48:47.6523689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6523872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6524249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6524419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6524439Z 
2022-11-23T01:48:47.6524548Z Running tests...
2022-11-23T01:48:47.6524811Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6525122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6525412Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6525628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10982
2022-11-23T01:48:47.6525848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10983
2022-11-23T01:48:47.6526216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6526371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6526747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6526934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6527299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6527529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6527906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6528099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6528343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6528584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6528961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6529352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6529577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6529806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6530061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7gx4bns
2022-11-23T01:48:47.6530332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7gx4bns/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6530581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nisg0fd
2022-11-23T01:48:47.6530843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nisg0fd/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6531612Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:47.6531720Z ok (6.333s)
2022-11-23T01:48:47.6531740Z 
2022-11-23T01:48:47.6532052Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6532156Z Ran 1 test in 6.333s
2022-11-23T01:48:47.6532175Z 
2022-11-23T01:48:47.6532270Z OK
2022-11-23T01:48:47.6532289Z 
2022-11-23T01:48:47.6532414Z Generating XML reports...
2022-11-23T01:48:47.6532866Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013449.xml
2022-11-23T01:48:47.6533236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6533412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6533792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6533986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6534005Z 
2022-11-23T01:48:47.6534097Z Running tests...
2022-11-23T01:48:47.6534366Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6534682Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6534958Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6535702Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.607s)
2022-11-23T01:48:47.6535724Z 
2022-11-23T01:48:47.6535983Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6536152Z Ran 1 test in 1.607s
2022-11-23T01:48:47.6536171Z 
2022-11-23T01:48:47.6536280Z OK (skipped=1)
2022-11-23T01:48:47.6536300Z 
2022-11-23T01:48:47.6536423Z Generating XML reports...
2022-11-23T01:48:47.6536874Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013458.xml
2022-11-23T01:48:47.6537225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6537399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6537774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6538180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6538203Z 
2022-11-23T01:48:47.6538319Z Running tests...
2022-11-23T01:48:47.6538586Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6538909Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6539236Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-11-23T01:48:47.6539259Z 
2022-11-23T01:48:47.6539515Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6539610Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6539630Z 
2022-11-23T01:48:47.6539739Z OK (skipped=1)
2022-11-23T01:48:47.6539758Z 
2022-11-23T01:48:47.6539880Z Generating XML reports...
2022-11-23T01:48:47.6540324Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013502.xml
2022-11-23T01:48:47.6540690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6540867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6541248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6541435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6541456Z 
2022-11-23T01:48:47.6541625Z Running tests...
2022-11-23T01:48:47.6541900Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6542212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6542458Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6543196Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.653s)
2022-11-23T01:48:47.6543221Z 
2022-11-23T01:48:47.6543478Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6543593Z Ran 1 test in 1.653s
2022-11-23T01:48:47.6543612Z 
2022-11-23T01:48:47.6543722Z OK (skipped=1)
2022-11-23T01:48:47.6543741Z 
2022-11-23T01:48:47.6543869Z Generating XML reports...
2022-11-23T01:48:47.6544314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013504.xml
2022-11-23T01:48:47.6544662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6544835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6545211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6545400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6545419Z 
2022-11-23T01:48:47.6545596Z Running tests...
2022-11-23T01:48:47.6545858Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6546167Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6546439Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6546638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11201
2022-11-23T01:48:47.6546855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11202
2022-11-23T01:48:47.6547221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6547394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6547767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6547957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6548321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6548497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6548875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6549045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6549286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6549526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6549923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6550316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6550548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6550821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6551082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rlrb15g
2022-11-23T01:48:47.6551349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rlrb15g/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6551580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nbenlk4
2022-11-23T01:48:47.6551845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nbenlk4/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6552633Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T01:48:47.6552967Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T01:48:47.6553749Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T01:48:47.6554077Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T01:48:47.6554185Z ok (7.256s)
2022-11-23T01:48:47.6554206Z 
2022-11-23T01:48:47.6554469Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6554584Z Ran 1 test in 7.256s
2022-11-23T01:48:47.6554651Z 
2022-11-23T01:48:47.6554748Z OK
2022-11-23T01:48:47.6554767Z 
2022-11-23T01:48:47.6554875Z Generating XML reports...
2022-11-23T01:48:47.6555328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013509.xml
2022-11-23T01:48:47.6555701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6555878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6556254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6556446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6556466Z 
2022-11-23T01:48:47.6556574Z Running tests...
2022-11-23T01:48:47.6556837Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6557149Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6557405Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6558151Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.599s)
2022-11-23T01:48:47.6558173Z 
2022-11-23T01:48:47.6558430Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6558544Z Ran 1 test in 1.599s
2022-11-23T01:48:47.6558563Z 
2022-11-23T01:48:47.6558672Z OK (skipped=1)
2022-11-23T01:48:47.6558691Z 
2022-11-23T01:48:47.6558817Z Generating XML reports...
2022-11-23T01:48:47.6559259Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013518.xml
2022-11-23T01:48:47.6559633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6559809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6560214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6560409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6560428Z 
2022-11-23T01:48:47.6560539Z Running tests...
2022-11-23T01:48:47.6560801Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6561113Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6561382Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6562117Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2022-11-23T01:48:47.6562142Z 
2022-11-23T01:48:47.6562406Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6562522Z Ran 1 test in 1.638s
2022-11-23T01:48:47.6562541Z 
2022-11-23T01:48:47.6562651Z OK (skipped=1)
2022-11-23T01:48:47.6562670Z 
2022-11-23T01:48:47.6562777Z Generating XML reports...
2022-11-23T01:48:47.6563220Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013522.xml
2022-11-23T01:48:47.6563586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6563758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6564132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6564377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6564398Z 
2022-11-23T01:48:47.6564510Z Running tests...
2022-11-23T01:48:47.6564778Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6565071Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6565359Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6565577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11418
2022-11-23T01:48:47.6565791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11419
2022-11-23T01:48:47.6566160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6566337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6566716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6566910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6567277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6567431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6567804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6567989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6568229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6568471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6568874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6569318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6569553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6569789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6569990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6570221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6570615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6571012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6571270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dveoa48
2022-11-23T01:48:47.6571540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dveoa48/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6571793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gn7umgb
2022-11-23T01:48:47.6572058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gn7umgb/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6572289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6572503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6572731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6573017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6573122Z ok (7.679s)
2022-11-23T01:48:47.6573142Z 
2022-11-23T01:48:47.6573411Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6573524Z Ran 1 test in 7.679s
2022-11-23T01:48:47.6573547Z 
2022-11-23T01:48:47.6573640Z OK
2022-11-23T01:48:47.6573659Z 
2022-11-23T01:48:47.6573786Z Generating XML reports...
2022-11-23T01:48:47.6574216Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013527.xml
2022-11-23T01:48:47.6574590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6574765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6575145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6575339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6575359Z 
2022-11-23T01:48:47.6575469Z Running tests...
2022-11-23T01:48:47.6575733Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6576048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6576319Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6576517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11539
2022-11-23T01:48:47.6576730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11540
2022-11-23T01:48:47.6577101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6577273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6577647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6577838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6578519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6578704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6579061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6579248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6579492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6579733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6580131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6580530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6580762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6581037Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:48:47.6581258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6581505Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:48:47.6581758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsalz170
2022-11-23T01:48:47.6582022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsalz170/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6582347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlctpr_l
2022-11-23T01:48:47.6582617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlctpr_l/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6582855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6583088Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6583319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6583548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6583806Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:48:47.6584083Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:48:47.6584361Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:48:47.6584631Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:48:47.6584865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6585092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6585321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6585548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6585818Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:48:47.6586069Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:48:47.6586346Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T01:48:47.6586611Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T01:48:47.6586888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6587127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6587359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6587584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6587689Z ok (8.025s)
2022-11-23T01:48:47.6587709Z 
2022-11-23T01:48:47.6587961Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6588077Z Ran 1 test in 8.025s
2022-11-23T01:48:47.6588097Z 
2022-11-23T01:48:47.6588196Z OK
2022-11-23T01:48:47.6588215Z 
2022-11-23T01:48:47.6588339Z Generating XML reports...
2022-11-23T01:48:47.6588794Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013537.xml
2022-11-23T01:48:47.6589166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6589345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6589722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6589913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6589933Z 
2022-11-23T01:48:47.6590024Z Running tests...
2022-11-23T01:48:47.6590287Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6590600Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6590960Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6591706Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2022-11-23T01:48:47.6591728Z 
2022-11-23T01:48:47.6591984Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6592099Z Ran 1 test in 1.637s
2022-11-23T01:48:47.6592119Z 
2022-11-23T01:48:47.6592229Z OK (skipped=1)
2022-11-23T01:48:47.6592248Z 
2022-11-23T01:48:47.6592374Z Generating XML reports...
2022-11-23T01:48:47.6592796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013547.xml
2022-11-23T01:48:47.6593163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6593339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6593718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6593905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6593924Z 
2022-11-23T01:48:47.6594033Z Running tests...
2022-11-23T01:48:47.6594290Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6594601Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6594868Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6595067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11692
2022-11-23T01:48:47.6595281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11693
2022-11-23T01:48:47.6595653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6595826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6596252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6596450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6596820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6596994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6597367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6597536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6597784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6598024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6598425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6598817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6599042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6599581Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:48:47.6599903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6600448Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:48:47.6600704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpceknywe2
2022-11-23T01:48:47.6600953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpceknywe2/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6601202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfza2z7fm
2022-11-23T01:48:47.6601466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfza2z7fm/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6601702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6601977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6602263Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T01:48:47.6602592Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T01:48:47.6602931Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T01:48:47.6603245Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T01:48:47.6603569Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T01:48:47.6603918Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T01:48:47.6604342Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T01:48:47.6604662Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T01:48:47.6604896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6605128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6605232Z ok (7.439s)
2022-11-23T01:48:47.6605251Z 
2022-11-23T01:48:47.6605526Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6605641Z Ran 1 test in 7.439s
2022-11-23T01:48:47.6605683Z 
2022-11-23T01:48:47.6605818Z OK
2022-11-23T01:48:47.6605852Z 
2022-11-23T01:48:47.6605959Z Generating XML reports...
2022-11-23T01:48:47.6606487Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013552.xml
2022-11-23T01:48:47.6606871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6607046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6607424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6607614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6607634Z 
2022-11-23T01:48:47.6607745Z Running tests...
2022-11-23T01:48:47.6608011Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6608306Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6608756Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6608777Z 
2022-11-23T01:48:47.6609092Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6609207Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6609231Z 
2022-11-23T01:48:47.6609341Z OK (skipped=1)
2022-11-23T01:48:47.6609360Z 
2022-11-23T01:48:47.6609485Z Generating XML reports...
2022-11-23T01:48:47.6609931Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013602.xml
2022-11-23T01:48:47.6610301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6610478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6610856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6611026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6611046Z 
2022-11-23T01:48:47.6611155Z Running tests...
2022-11-23T01:48:47.6611421Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6611734Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6612117Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6612137Z 
2022-11-23T01:48:47.6612393Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6612507Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6612526Z 
2022-11-23T01:48:47.6612635Z OK (skipped=1)
2022-11-23T01:48:47.6612658Z 
2022-11-23T01:48:47.6612781Z Generating XML reports...
2022-11-23T01:48:47.6613208Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013604.xml
2022-11-23T01:48:47.6613626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6613807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6614181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6614370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6614389Z 
2022-11-23T01:48:47.6614498Z Running tests...
2022-11-23T01:48:47.6614756Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6615067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6615515Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6615535Z 
2022-11-23T01:48:47.6615777Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6615890Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6615909Z 
2022-11-23T01:48:47.6616018Z OK (skipped=1)
2022-11-23T01:48:47.6616037Z 
2022-11-23T01:48:47.6616161Z Generating XML reports...
2022-11-23T01:48:47.6616598Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013606.xml
2022-11-23T01:48:47.6616965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6617139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6617573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6617759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6617779Z 
2022-11-23T01:48:47.6617871Z Running tests...
2022-11-23T01:48:47.6618376Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6618690Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6619128Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6619149Z 
2022-11-23T01:48:47.6619400Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6619514Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6619538Z 
2022-11-23T01:48:47.6619647Z OK (skipped=1)
2022-11-23T01:48:47.6619666Z 
2022-11-23T01:48:47.6619789Z Generating XML reports...
2022-11-23T01:48:47.6620228Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013609.xml
2022-11-23T01:48:47.6620580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6620753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6621124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6621310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6621328Z 
2022-11-23T01:48:47.6621435Z Running tests...
2022-11-23T01:48:47.6621688Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6621997Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6622526Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6622549Z 
2022-11-23T01:48:47.6622825Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6622920Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6622957Z 
2022-11-23T01:48:47.6623049Z OK (skipped=1)
2022-11-23T01:48:47.6623067Z 
2022-11-23T01:48:47.6623191Z Generating XML reports...
2022-11-23T01:48:47.6623632Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013611.xml
2022-11-23T01:48:47.6623996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6624170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6624545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6624731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6624750Z 
2022-11-23T01:48:47.6624864Z Running tests...
2022-11-23T01:48:47.6625102Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6625411Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6625847Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6625867Z 
2022-11-23T01:48:47.6626128Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6626313Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6626332Z 
2022-11-23T01:48:47.6626441Z OK (skipped=1)
2022-11-23T01:48:47.6626460Z 
2022-11-23T01:48:47.6626587Z Generating XML reports...
2022-11-23T01:48:47.6627039Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013614.xml
2022-11-23T01:48:47.6627403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6627558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6627936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6628125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6628145Z 
2022-11-23T01:48:47.6628252Z Running tests...
2022-11-23T01:48:47.6628514Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6628831Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6629276Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6629297Z 
2022-11-23T01:48:47.6629553Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6629667Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6629686Z 
2022-11-23T01:48:47.6629794Z OK (skipped=1)
2022-11-23T01:48:47.6629814Z 
2022-11-23T01:48:47.6629920Z Generating XML reports...
2022-11-23T01:48:47.6630371Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013616.xml
2022-11-23T01:48:47.6630736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6630915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6631292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6631532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6631556Z 
2022-11-23T01:48:47.6631672Z Running tests...
2022-11-23T01:48:47.6631931Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6632223Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6632658Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6632679Z 
2022-11-23T01:48:47.6632941Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6633058Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6633078Z 
2022-11-23T01:48:47.6633185Z OK (skipped=1)
2022-11-23T01:48:47.6633204Z 
2022-11-23T01:48:47.6633325Z Generating XML reports...
2022-11-23T01:48:47.6633774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013618.xml
2022-11-23T01:48:47.6634143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6634319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6634697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6634870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6634890Z 
2022-11-23T01:48:47.6634999Z Running tests...
2022-11-23T01:48:47.6635260Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6635633Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6636076Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6636096Z 
2022-11-23T01:48:47.6636358Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6636471Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6636491Z 
2022-11-23T01:48:47.6636602Z OK (skipped=1)
2022-11-23T01:48:47.6636621Z 
2022-11-23T01:48:47.6636745Z Generating XML reports...
2022-11-23T01:48:47.6637172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013621.xml
2022-11-23T01:48:47.6637540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6637717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6638096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6638288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6638307Z 
2022-11-23T01:48:47.6638418Z Running tests...
2022-11-23T01:48:47.6638679Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6638992Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6639429Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6639450Z 
2022-11-23T01:48:47.6639694Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6639810Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6639829Z 
2022-11-23T01:48:47.6639940Z OK (skipped=1)
2022-11-23T01:48:47.6639959Z 
2022-11-23T01:48:47.6640085Z Generating XML reports...
2022-11-23T01:48:47.6640597Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013623.xml
2022-11-23T01:48:47.6640976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6641149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6641523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6641709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6641729Z 
2022-11-23T01:48:47.6641819Z Running tests...
2022-11-23T01:48:47.6642084Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6642394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6642780Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6642800Z 
2022-11-23T01:48:47.6643057Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6643169Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6643188Z 
2022-11-23T01:48:47.6643296Z OK (skipped=1)
2022-11-23T01:48:47.6643315Z 
2022-11-23T01:48:47.6643439Z Generating XML reports...
2022-11-23T01:48:47.6643878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013626.xml
2022-11-23T01:48:47.6644226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6644458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6644833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6645024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6645044Z 
2022-11-23T01:48:47.6645155Z Running tests...
2022-11-23T01:48:47.6645416Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6645726Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6646109Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:48:47.6646129Z 
2022-11-23T01:48:47.6646387Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6646487Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6646523Z 
2022-11-23T01:48:47.6646614Z OK (skipped=1)
2022-11-23T01:48:47.6646633Z 
2022-11-23T01:48:47.6646755Z Generating XML reports...
2022-11-23T01:48:47.6647199Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013628.xml
2022-11-23T01:48:47.6647568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6647743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6648114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6648301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6648321Z 
2022-11-23T01:48:47.6648429Z Running tests...
2022-11-23T01:48:47.6648669Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6648984Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6649246Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6650036Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.618s)
2022-11-23T01:48:47.6650058Z 
2022-11-23T01:48:47.6650329Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6650443Z Ran 1 test in 1.618s
2022-11-23T01:48:47.6650463Z 
2022-11-23T01:48:47.6650572Z OK (skipped=1)
2022-11-23T01:48:47.6650591Z 
2022-11-23T01:48:47.6650714Z Generating XML reports...
2022-11-23T01:48:47.6651159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013630.xml
2022-11-23T01:48:47.6651517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6651699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6652078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6652267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6652287Z 
2022-11-23T01:48:47.6652396Z Running tests...
2022-11-23T01:48:47.6652653Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6652963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6653212Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6653490Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12241
2022-11-23T01:48:47.6653686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12242
2022-11-23T01:48:47.6654057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6654231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6654609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6654798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6655159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6655332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6655702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6655876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6656119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6656362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6656765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6657162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6657389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6657612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6657865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrlchtaf
2022-11-23T01:48:47.6658361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrlchtaf/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6658594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0k3an6sl
2022-11-23T01:48:47.6658937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0k3an6sl/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6659051Z ok (6.691s)
2022-11-23T01:48:47.6659072Z 
2022-11-23T01:48:47.6659348Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6659463Z Ran 1 test in 6.691s
2022-11-23T01:48:47.6659483Z 
2022-11-23T01:48:47.6659579Z OK
2022-11-23T01:48:47.6659598Z 
2022-11-23T01:48:47.6659724Z Generating XML reports...
2022-11-23T01:48:47.6660173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013634.xml
2022-11-23T01:48:47.6660539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6660702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6661079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6661273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6661292Z 
2022-11-23T01:48:47.6661403Z Running tests...
2022-11-23T01:48:47.6661666Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6661980Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6662249Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6662466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12355
2022-11-23T01:48:47.6662662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12356
2022-11-23T01:48:47.6663111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6663288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6663669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6663859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6664223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6664399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6664772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6664959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6665186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6665426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6665832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6666228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6666457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6666684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6666937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiautc8i4
2022-11-23T01:48:47.6667197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiautc8i4/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6667443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7o0eo7qc
2022-11-23T01:48:47.6667687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7o0eo7qc/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6667963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6668208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6668611Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpiautc8i4'>
2022-11-23T01:48:47.6668778Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T01:48:47.6669174Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp7o0eo7qc'>
2022-11-23T01:48:47.6669338Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T01:48:47.6669442Z ok (7.362s)
2022-11-23T01:48:47.6669466Z 
2022-11-23T01:48:47.6669714Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6669829Z Ran 1 test in 7.362s
2022-11-23T01:48:47.6669849Z 
2022-11-23T01:48:47.6669941Z OK
2022-11-23T01:48:47.6669961Z 
2022-11-23T01:48:47.6670084Z Generating XML reports...
2022-11-23T01:48:47.6670536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013644.xml
2022-11-23T01:48:47.6670907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6671082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6671459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6671627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6671665Z 
2022-11-23T01:48:47.6671818Z Running tests...
2022-11-23T01:48:47.6672087Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6672400Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6672678Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-11-23T01:48:47.6672698Z 
2022-11-23T01:48:47.6672954Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6673067Z Ran 1 test in 0.006s
2022-11-23T01:48:47.6673086Z 
2022-11-23T01:48:47.6673197Z OK (skipped=1)
2022-11-23T01:48:47.6673216Z 
2022-11-23T01:48:47.6673339Z Generating XML reports...
2022-11-23T01:48:47.6673763Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013654.xml
2022-11-23T01:48:47.6674131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6674310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6674689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6674881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6674900Z 
2022-11-23T01:48:47.6675009Z Running tests...
2022-11-23T01:48:47.6675270Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6675583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6675845Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6676044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12507
2022-11-23T01:48:47.6676257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12508
2022-11-23T01:48:47.6676631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6676805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6677231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6677428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6677797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6677971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6678326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6678514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6678753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6678998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6679403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6679798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6680024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6680249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6680502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5nyaayck
2022-11-23T01:48:47.6680752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5nyaayck/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6681001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpciez3j0z
2022-11-23T01:48:47.6681323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpciez3j0z/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6681560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6681793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6681901Z ok (7.353s)
2022-11-23T01:48:47.6681920Z 
2022-11-23T01:48:47.6682190Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6682306Z Ran 1 test in 7.353s
2022-11-23T01:48:47.6682326Z 
2022-11-23T01:48:47.6682400Z OK
2022-11-23T01:48:47.6682438Z 
2022-11-23T01:48:47.6682544Z Generating XML reports...
2022-11-23T01:48:47.6682992Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013656.xml
2022-11-23T01:48:47.6683359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6683538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6683917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6684107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6684126Z 
2022-11-23T01:48:47.6684234Z Running tests...
2022-11-23T01:48:47.6684498Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6684792Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6685074Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6685290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12626
2022-11-23T01:48:47.6685504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12627
2022-11-23T01:48:47.6685879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6686054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6686479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6686678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6687048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6687201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6687575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6687768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6688014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6688254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6688653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6689046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6689273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6689478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6689715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6689951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6690404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6690795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6691037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:47.6691278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:47.6691663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.6692048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.6692301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpraim_h88
2022-11-23T01:48:47.6692554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpraim_h88/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6692807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprkk2wqcq
2022-11-23T01:48:47.6693074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprkk2wqcq/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6693177Z ok (5.854s)
2022-11-23T01:48:47.6693197Z 
2022-11-23T01:48:47.6693464Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6693578Z Ran 1 test in 5.854s
2022-11-23T01:48:47.6693597Z 
2022-11-23T01:48:47.6693689Z OK
2022-11-23T01:48:47.6693709Z 
2022-11-23T01:48:47.6693833Z Generating XML reports...
2022-11-23T01:48:47.6694264Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013706.xml
2022-11-23T01:48:47.6694635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6694816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6695195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6695433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6695454Z 
2022-11-23T01:48:47.6695568Z Running tests...
2022-11-23T01:48:47.6695838Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6696154Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6696433Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6696633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12748
2022-11-23T01:48:47.6696849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12749
2022-11-23T01:48:47.6697222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6697397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6697775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6698178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6698559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6698733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6699090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6699279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6699608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6699884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6700293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6700688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6700919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6701144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6701381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6701598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6701995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6702386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6702626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:47.6702861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:47.6703246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.6703635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.6703889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27m3ubh3
2022-11-23T01:48:47.6704160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27m3ubh3/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6704392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasrgmn58
2022-11-23T01:48:47.6704731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasrgmn58/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6704845Z ok (16.731s)
2022-11-23T01:48:47.6704865Z 
2022-11-23T01:48:47.6705135Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6705250Z Ran 1 test in 16.731s
2022-11-23T01:48:47.6705269Z 
2022-11-23T01:48:47.6705366Z OK
2022-11-23T01:48:47.6705385Z 
2022-11-23T01:48:47.6705510Z Generating XML reports...
2022-11-23T01:48:47.6705964Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013714.xml
2022-11-23T01:48:47.6706335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6706499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6706879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6707071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6707091Z 
2022-11-23T01:48:47.6707201Z Running tests...
2022-11-23T01:48:47.6707462Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6707776Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6708083Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6708301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12870
2022-11-23T01:48:47.6708497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12871
2022-11-23T01:48:47.6708958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6709131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6709511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6709699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6710062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6710238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6710612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6710799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6711029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6711270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6711670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6712066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6712294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6712516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6712769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmfcy19r
2022-11-23T01:48:47.6713035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmfcy19r/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6713288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3johaf1z
2022-11-23T01:48:47.6713537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3johaf1z/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6713641Z ok (7.778s)
2022-11-23T01:48:47.6713711Z 
2022-11-23T01:48:47.6713992Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6714108Z Ran 1 test in 7.779s
2022-11-23T01:48:47.6714128Z 
2022-11-23T01:48:47.6714218Z OK
2022-11-23T01:48:47.6714238Z 
2022-11-23T01:48:47.6714361Z Generating XML reports...
2022-11-23T01:48:47.6714813Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013734.xml
2022-11-23T01:48:47.6715182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6715339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6715723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6715911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6715931Z 
2022-11-23T01:48:47.6716043Z Running tests...
2022-11-23T01:48:47.6716305Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6716623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6716912Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6717129Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12989
2022-11-23T01:48:47.6717343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12990
2022-11-23T01:48:47.6717694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6717925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6718305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6718497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6718865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6719038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6719414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6719604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6719827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6720071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6720471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6720867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6721093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6721317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6721570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8j12bdc
2022-11-23T01:48:47.6721833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8j12bdc/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6722080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cgqontc
2022-11-23T01:48:47.6722331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cgqontc/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6722435Z ok (7.771s)
2022-11-23T01:48:47.6722455Z 
2022-11-23T01:48:47.6722721Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6722881Z Ran 1 test in 7.771s
2022-11-23T01:48:47.6722902Z 
2022-11-23T01:48:47.6722998Z OK
2022-11-23T01:48:47.6723017Z 
2022-11-23T01:48:47.6723146Z Generating XML reports...
2022-11-23T01:48:47.6723595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013744.xml
2022-11-23T01:48:47.6723965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6724140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6724503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6724697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6724716Z 
2022-11-23T01:48:47.6724825Z Running tests...
2022-11-23T01:48:47.6725087Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6725402Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6725657Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6725874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13108
2022-11-23T01:48:47.6726090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13109
2022-11-23T01:48:47.6726438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6726611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6727047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6727235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6727604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6727778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6728153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6728340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6728561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6728805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6729199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6729597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6729825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6730052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6730308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2vt5gbi
2022-11-23T01:48:47.6730573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2vt5gbi/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6730822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_kguiiz
2022-11-23T01:48:47.6731067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_kguiiz/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6731176Z ok (7.665s)
2022-11-23T01:48:47.6731195Z 
2022-11-23T01:48:47.6731459Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6731571Z Ran 1 test in 7.665s
2022-11-23T01:48:47.6731590Z 
2022-11-23T01:48:47.6731682Z OK
2022-11-23T01:48:47.6731701Z 
2022-11-23T01:48:47.6731906Z Generating XML reports...
2022-11-23T01:48:47.6732364Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013754.xml
2022-11-23T01:48:47.6732732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6732904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6733263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6733450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6733474Z 
2022-11-23T01:48:47.6733582Z Running tests...
2022-11-23T01:48:47.6733844Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6734159Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6734423Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6734642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13223
2022-11-23T01:48:47.6734858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13224
2022-11-23T01:48:47.6735206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6735383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6735765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6736013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6736382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6736556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6736939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6737128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6737366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6737587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6738214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6738620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6738855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6739086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6739344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewpzy6o7
2022-11-23T01:48:47.6739613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewpzy6o7/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6739865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxu212hf
2022-11-23T01:48:47.6740125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxu212hf/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6740968Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:47.6741755Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:47.6741994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6742213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6742318Z ok (7.307s)
2022-11-23T01:48:47.6742339Z 
2022-11-23T01:48:47.6742615Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6742731Z Ran 1 test in 7.307s
2022-11-23T01:48:47.6742750Z 
2022-11-23T01:48:47.6742844Z OK
2022-11-23T01:48:47.6742863Z 
2022-11-23T01:48:47.6742988Z Generating XML reports...
2022-11-23T01:48:47.6743433Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013804.xml
2022-11-23T01:48:47.6743802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6743975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6744332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6744594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6744613Z 
2022-11-23T01:48:47.6744726Z Running tests...
2022-11-23T01:48:47.6744995Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6745308Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6745586Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6746332Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-11-23T01:48:47.6746354Z 
2022-11-23T01:48:47.6746619Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6746738Z Ran 1 test in 1.613s
2022-11-23T01:48:47.6746757Z 
2022-11-23T01:48:47.6746848Z OK (skipped=1)
2022-11-23T01:48:47.6746887Z 
2022-11-23T01:48:47.6746993Z Generating XML reports...
2022-11-23T01:48:47.6747442Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013814.xml
2022-11-23T01:48:47.6747811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6747983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6748361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6748550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6748570Z 
2022-11-23T01:48:47.6748683Z Running tests...
2022-11-23T01:48:47.6748945Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6749248Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6749530Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6750321Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.600s)
2022-11-23T01:48:47.6750344Z 
2022-11-23T01:48:47.6750616Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6750730Z Ran 1 test in 1.600s
2022-11-23T01:48:47.6750750Z 
2022-11-23T01:48:47.6750861Z OK (skipped=1)
2022-11-23T01:48:47.6750880Z 
2022-11-23T01:48:47.6751004Z Generating XML reports...
2022-11-23T01:48:47.6751452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013818.xml
2022-11-23T01:48:47.6751832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6752010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6752372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6752564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6752583Z 
2022-11-23T01:48:47.6752694Z Running tests...
2022-11-23T01:48:47.6752953Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6753263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6753538Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6753811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13410
2022-11-23T01:48:47.6754027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13411
2022-11-23T01:48:47.6754381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6754558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6754937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6755127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6755494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6755666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6756040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6756232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6756469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6756696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6757095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6757488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6757717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6757944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6758198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehfd44_b
2022-11-23T01:48:47.6758471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehfd44_b/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6758772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9p7nrmt
2022-11-23T01:48:47.6759048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9p7nrmt/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6759367Z STAGE:2022-11-23 01:38:28 13410:13410 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6759688Z STAGE:2022-11-23 01:38:28 13411:13411 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6759924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6760157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6760491Z STAGE:2022-11-23 01:38:29 13410:13410 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6760822Z STAGE:2022-11-23 01:38:29 13411:13411 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6761171Z STAGE:2022-11-23 01:38:29 13410:13410 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6761518Z STAGE:2022-11-23 01:38:29 13411:13411 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6762288Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:47.6763116Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:47.6763657Z STAGE:2022-11-23 01:38:29 13410:13410 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:38:29 13411:13411 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6763678Z 
2022-11-23T01:48:47.6764013Z STAGE:2022-11-23 01:38:29 13410:13410 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6764339Z STAGE:2022-11-23 01:38:29 13411:13411 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6764670Z STAGE:2022-11-23 01:38:29 13410:13410 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6765014Z STAGE:2022-11-23 01:38:29 13411:13411 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6765118Z ok (6.772s)
2022-11-23T01:48:47.6765138Z 
2022-11-23T01:48:47.6765403Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6765519Z Ran 1 test in 6.772s
2022-11-23T01:48:47.6765539Z 
2022-11-23T01:48:47.6765632Z OK
2022-11-23T01:48:47.6765651Z 
2022-11-23T01:48:47.6765776Z Generating XML reports...
2022-11-23T01:48:47.6766225Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013822.xml
2022-11-23T01:48:47.6766577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6766758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6767137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6767329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6767417Z 
2022-11-23T01:48:47.6767539Z Running tests...
2022-11-23T01:48:47.6767805Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6768118Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6768388Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6768608Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13532
2022-11-23T01:48:47.6768806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13533
2022-11-23T01:48:47.6769173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6769350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6769733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6769922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6770287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6770457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6770834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6771005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6771246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6771559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6771957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6772356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6772588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6772813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6773072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftj2ktne
2022-11-23T01:48:47.6773342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftj2ktne/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6773574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qsdccgy
2022-11-23T01:48:47.6773845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qsdccgy/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6773947Z ok (6.965s)
2022-11-23T01:48:47.6773967Z 
2022-11-23T01:48:47.6774234Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6774348Z Ran 1 test in 6.965s
2022-11-23T01:48:47.6774368Z 
2022-11-23T01:48:47.6774460Z OK
2022-11-23T01:48:47.6774478Z 
2022-11-23T01:48:47.6774602Z Generating XML reports...
2022-11-23T01:48:47.6775050Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013832.xml
2022-11-23T01:48:47.6775417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6775575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6775954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6776145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6776165Z 
2022-11-23T01:48:47.6776273Z Running tests...
2022-11-23T01:48:47.6776585Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6776907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6777183Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6778146Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2022-11-23T01:48:47.6778170Z 
2022-11-23T01:48:47.6778445Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6778548Z Ran 1 test in 1.641s
2022-11-23T01:48:47.6778587Z 
2022-11-23T01:48:47.6778680Z OK (skipped=1)
2022-11-23T01:48:47.6778699Z 
2022-11-23T01:48:47.6778826Z Generating XML reports...
2022-11-23T01:48:47.6779279Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013841.xml
2022-11-23T01:48:47.6779649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6779823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6780201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6780389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6780409Z 
2022-11-23T01:48:47.6780517Z Running tests...
2022-11-23T01:48:47.6780757Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6781162Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6781445Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6781661Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13681
2022-11-23T01:48:47.6781875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13682
2022-11-23T01:48:47.6782238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6782411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6782786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6782955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6783325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6783498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6783875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6784063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6784307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6784552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6784951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6785345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6785559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6785782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6786093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mo3ad3c
2022-11-23T01:48:47.6786371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mo3ad3c/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6786622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8fnomt7
2022-11-23T01:48:47.6786889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8fnomt7/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6787799Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T01:48:47.6787921Z   warnings.warn(
2022-11-23T01:48:47.6788826Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T01:48:47.6788939Z   warnings.warn(
2022-11-23T01:48:47.6789175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6789389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6789491Z ok (6.355s)
2022-11-23T01:48:47.6789511Z 
2022-11-23T01:48:47.6789777Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6789946Z Ran 1 test in 6.355s
2022-11-23T01:48:47.6789966Z 
2022-11-23T01:48:47.6790059Z OK
2022-11-23T01:48:47.6790078Z 
2022-11-23T01:48:47.6790197Z Generating XML reports...
2022-11-23T01:48:47.6790659Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013846.xml
2022-11-23T01:48:47.6791028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6791203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6791579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6791767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6791787Z 
2022-11-23T01:48:47.6791879Z Running tests...
2022-11-23T01:48:47.6792141Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6792459Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6792732Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6793475Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2022-11-23T01:48:47.6793497Z 
2022-11-23T01:48:47.6793760Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6793872Z Ran 1 test in 1.645s
2022-11-23T01:48:47.6793892Z 
2022-11-23T01:48:47.6794000Z OK (skipped=1)
2022-11-23T01:48:47.6794019Z 
2022-11-23T01:48:47.6794141Z Generating XML reports...
2022-11-23T01:48:47.6794589Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013854.xml
2022-11-23T01:48:47.6794939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6795164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6795551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6795739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6795759Z 
2022-11-23T01:48:47.6795868Z Running tests...
2022-11-23T01:48:47.6796129Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6796442Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6796709Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6796912Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13833
2022-11-23T01:48:47.6797125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13834
2022-11-23T01:48:47.6797494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6797668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6798042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6798229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6798591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6798765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6799137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6799362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6799610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6799886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6800291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6800687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6800913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6801137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6801395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_oh9qgn
2022-11-23T01:48:47.6801658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_oh9qgn/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6801893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kxhx_8u
2022-11-23T01:48:47.6802155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kxhx_8u/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6802492Z STAGE:2022-11-23 01:39:04 13834:13834 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6802815Z STAGE:2022-11-23 01:39:04 13833:13833 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6803433Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.6803553Z   warnings.warn(
2022-11-23T01:48:47.6804167Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:48:47.6804327Z   warnings.warn(
2022-11-23T01:48:47.6804569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6804783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:47.6805127Z STAGE:2022-11-23 01:39:04 13833:13833 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6805453Z STAGE:2022-11-23 01:39:04 13834:13834 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6805797Z STAGE:2022-11-23 01:39:04 13833:13833 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6806140Z STAGE:2022-11-23 01:39:04 13834:13834 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6806473Z STAGE:2022-11-23 01:39:04 13833:13833 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6806807Z STAGE:2022-11-23 01:39:05 13833:13833 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6807149Z STAGE:2022-11-23 01:39:05 13833:13833 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6807253Z ok (7.168s)
2022-11-23T01:48:47.6807273Z 
2022-11-23T01:48:47.6807519Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6807631Z Ran 1 test in 7.169s
2022-11-23T01:48:47.6807650Z 
2022-11-23T01:48:47.6807743Z OK
2022-11-23T01:48:47.6807762Z 
2022-11-23T01:48:47.6807886Z Generating XML reports...
2022-11-23T01:48:47.6808333Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013859.xml
2022-11-23T01:48:47.6808759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6808935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6809315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6809487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6809524Z 
2022-11-23T01:48:47.6809615Z Running tests...
2022-11-23T01:48:47.6809874Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6810185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6810447Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6810663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13955
2022-11-23T01:48:47.6810883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13956
2022-11-23T01:48:47.6811250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6811425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6811782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6811967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6812328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6812501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6812876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6813063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6813309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6813550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6813979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6814381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6814608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6814832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6815087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadg1qav2
2022-11-23T01:48:47.6815355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadg1qav2/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6815609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1v13l7v
2022-11-23T01:48:47.6815873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1v13l7v/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6815976Z ok (6.958s)
2022-11-23T01:48:47.6815997Z 
2022-11-23T01:48:47.6816248Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6816362Z Ran 1 test in 6.958s
2022-11-23T01:48:47.6816382Z 
2022-11-23T01:48:47.6816474Z OK
2022-11-23T01:48:47.6816493Z 
2022-11-23T01:48:47.6816616Z Generating XML reports...
2022-11-23T01:48:47.6817064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013908.xml
2022-11-23T01:48:47.6817433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6817662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6818275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6818475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6818497Z 
2022-11-23T01:48:47.6818587Z Running tests...
2022-11-23T01:48:47.6818857Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6819170Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6819442Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6819657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14070
2022-11-23T01:48:47.6819871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14071
2022-11-23T01:48:47.6820236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6820415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6820775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6820964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6821330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6821504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6821878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6822066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6822305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6822549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6822949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6823400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6823639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6823864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6824118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxw33mc7
2022-11-23T01:48:47.6824385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxw33mc7/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6824633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9o0_ks6d
2022-11-23T01:48:47.6824897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9o0_ks6d/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6825003Z ok (7.058s)
2022-11-23T01:48:47.6825023Z 
2022-11-23T01:48:47.6825277Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6825391Z Ran 1 test in 7.058s
2022-11-23T01:48:47.6825410Z 
2022-11-23T01:48:47.6825504Z OK
2022-11-23T01:48:47.6825523Z 
2022-11-23T01:48:47.6825648Z Generating XML reports...
2022-11-23T01:48:47.6826094Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013918.xml
2022-11-23T01:48:47.6826460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6826636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6827010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6827265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6827285Z 
2022-11-23T01:48:47.6827375Z Running tests...
2022-11-23T01:48:47.6827647Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6827961Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6828236Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6828979Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.779s)
2022-11-23T01:48:47.6829002Z 
2022-11-23T01:48:47.6829259Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6829378Z Ran 1 test in 1.779s
2022-11-23T01:48:47.6829397Z 
2022-11-23T01:48:47.6829506Z OK (skipped=1)
2022-11-23T01:48:47.6829525Z 
2022-11-23T01:48:47.6829649Z Generating XML reports...
2022-11-23T01:48:47.6830098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013927.xml
2022-11-23T01:48:47.6830449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6830624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6831002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6831191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6831210Z 
2022-11-23T01:48:47.6831318Z Running tests...
2022-11-23T01:48:47.6831584Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6831898Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6832152Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6832957Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.660s)
2022-11-23T01:48:47.6832980Z 
2022-11-23T01:48:47.6833231Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6833344Z Ran 1 test in 1.660s
2022-11-23T01:48:47.6833363Z 
2022-11-23T01:48:47.6833470Z OK (skipped=1)
2022-11-23T01:48:47.6833489Z 
2022-11-23T01:48:47.6833612Z Generating XML reports...
2022-11-23T01:48:47.6834061Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013932.xml
2022-11-23T01:48:47.6834428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6834605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6834984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6835173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6835193Z 
2022-11-23T01:48:47.6835285Z Running tests...
2022-11-23T01:48:47.6835547Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6835859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6836144Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6836940Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.595s)
2022-11-23T01:48:47.6836961Z 
2022-11-23T01:48:47.6837217Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6837329Z Ran 1 test in 1.595s
2022-11-23T01:48:47.6837348Z 
2022-11-23T01:48:47.6837455Z OK (skipped=1)
2022-11-23T01:48:47.6837476Z 
2022-11-23T01:48:47.6837601Z Generating XML reports...
2022-11-23T01:48:47.6838024Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013936.xml
2022-11-23T01:48:47.6838392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6838569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6838941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6839132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6839152Z 
2022-11-23T01:48:47.6839260Z Running tests...
2022-11-23T01:48:47.6839516Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6839826Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6840117Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6840316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14287
2022-11-23T01:48:47.6840532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14288
2022-11-23T01:48:47.6840908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6841083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6841511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6841704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6842068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6842239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6842612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6842780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6843023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6843268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6843666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6844062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6844291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6844516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6844769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41yt4sog
2022-11-23T01:48:47.6845017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41yt4sog/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6845323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpup8c0fem
2022-11-23T01:48:47.6845587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpup8c0fem/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6845689Z ok (7.388s)
2022-11-23T01:48:47.6845710Z 
2022-11-23T01:48:47.6845981Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6846094Z Ran 1 test in 7.388s
2022-11-23T01:48:47.6846113Z 
2022-11-23T01:48:47.6846208Z OK
2022-11-23T01:48:47.6846227Z 
2022-11-23T01:48:47.6846350Z Generating XML reports...
2022-11-23T01:48:47.6846796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013940.xml
2022-11-23T01:48:47.6847147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6847321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6847698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6847894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6847914Z 
2022-11-23T01:48:47.6848021Z Running tests...
2022-11-23T01:48:47.6848287Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6848602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6848867Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6849068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14406
2022-11-23T01:48:47.6849283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14407
2022-11-23T01:48:47.6849652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6849831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6850206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6850442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6850818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6850991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6851368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6851537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6851779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6852021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6852424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6852820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6853045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6853422Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T01:48:47.6853677Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T01:48:47.6853903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6854262Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T01:48:47.6854515Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T01:48:47.6854828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1klzedw
2022-11-23T01:48:47.6855096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1klzedw/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6855348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvub5f8c
2022-11-23T01:48:47.6855614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvub5f8c/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6855717Z ok (6.959s)
2022-11-23T01:48:47.6855737Z 
2022-11-23T01:48:47.6856003Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6856099Z Ran 1 test in 6.959s
2022-11-23T01:48:47.6856135Z 
2022-11-23T01:48:47.6856211Z OK
2022-11-23T01:48:47.6856230Z 
2022-11-23T01:48:47.6856353Z Generating XML reports...
2022-11-23T01:48:47.6856802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013950.xml
2022-11-23T01:48:47.6857174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6857348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6857730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6858139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6858163Z 
2022-11-23T01:48:47.6858278Z Running tests...
2022-11-23T01:48:47.6858530Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6858842Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6859099Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6859315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14521
2022-11-23T01:48:47.6859534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14522
2022-11-23T01:48:47.6859901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6860154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6860542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6860711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6861076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6861253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6861625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6861816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6862057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6862300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6862698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6863090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6863301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6863535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6863755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6864053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6864446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6864836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6864939Z ok (4.356s)
2022-11-23T01:48:47.6864959Z 
2022-11-23T01:48:47.6865220Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6865332Z Ran 1 test in 4.356s
2022-11-23T01:48:47.6865351Z 
2022-11-23T01:48:47.6865426Z OK
2022-11-23T01:48:47.6865445Z 
2022-11-23T01:48:47.6865570Z Generating XML reports...
2022-11-23T01:48:47.6866020Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013959.xml
2022-11-23T01:48:47.6866393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6866569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6866951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6867138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6867157Z 
2022-11-23T01:48:47.6867265Z Running tests...
2022-11-23T01:48:47.6867511Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6867824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6868072Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6868287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14628
2022-11-23T01:48:47.6868499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14629
2022-11-23T01:48:47.6868872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6869047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6869473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6869666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6870014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6870186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6870557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6870746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6870992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6871236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6871636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6872030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6872256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6872473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6872692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6872921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6873398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6873791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6873895Z ok (4.456s)
2022-11-23T01:48:47.6873915Z 
2022-11-23T01:48:47.6874177Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6874289Z Ran 1 test in 4.457s
2022-11-23T01:48:47.6874308Z 
2022-11-23T01:48:47.6874384Z OK
2022-11-23T01:48:47.6874421Z 
2022-11-23T01:48:47.6874526Z Generating XML reports...
2022-11-23T01:48:47.6874974Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014006.xml
2022-11-23T01:48:47.6875344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6875524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6875901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6876093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6876113Z 
2022-11-23T01:48:47.6876221Z Running tests...
2022-11-23T01:48:47.6876482Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6876774Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6877046Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6877790Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.649s)
2022-11-23T01:48:47.6877817Z 
2022-11-23T01:48:47.6878078Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6878190Z Ran 1 test in 1.649s
2022-11-23T01:48:47.6878210Z 
2022-11-23T01:48:47.6878372Z OK (skipped=1)
2022-11-23T01:48:47.6878393Z 
2022-11-23T01:48:47.6878520Z Generating XML reports...
2022-11-23T01:48:47.6878968Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014013.xml
2022-11-23T01:48:47.6879336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6879511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6879871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6880065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6880084Z 
2022-11-23T01:48:47.6880193Z Running tests...
2022-11-23T01:48:47.6880453Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6880771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6881045Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6881780Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.643s)
2022-11-23T01:48:47.6881801Z 
2022-11-23T01:48:47.6882062Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6882174Z Ran 1 test in 1.643s
2022-11-23T01:48:47.6882241Z 
2022-11-23T01:48:47.6882336Z OK (skipped=1)
2022-11-23T01:48:47.6882372Z 
2022-11-23T01:48:47.6882478Z Generating XML reports...
2022-11-23T01:48:47.6882927Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014017.xml
2022-11-23T01:48:47.6883296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6883471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6883850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6884038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6884058Z 
2022-11-23T01:48:47.6884167Z Running tests...
2022-11-23T01:48:47.6884425Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6884719Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6884991Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6885212Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14803
2022-11-23T01:48:47.6885426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14804
2022-11-23T01:48:47.6885790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6885963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6886339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6886527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6886874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6887050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6887425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6887660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6887906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6888147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6888546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6888942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6889169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6889381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6889484Z ok (4.215s)
2022-11-23T01:48:47.6889504Z 
2022-11-23T01:48:47.6889772Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6889886Z Ran 1 test in 4.215s
2022-11-23T01:48:47.6889905Z 
2022-11-23T01:48:47.6889997Z OK
2022-11-23T01:48:47.6890016Z 
2022-11-23T01:48:47.6890143Z Generating XML reports...
2022-11-23T01:48:47.6890592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014022.xml
2022-11-23T01:48:47.6890964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6891138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6891494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6891738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6891758Z 
2022-11-23T01:48:47.6891868Z Running tests...
2022-11-23T01:48:47.6892138Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6892450Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6892697Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6892717Z 
2022-11-23T01:48:47.6892976Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6893087Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6893106Z 
2022-11-23T01:48:47.6893197Z OK (skipped=1)
2022-11-23T01:48:47.6893233Z 
2022-11-23T01:48:47.6893338Z Generating XML reports...
2022-11-23T01:48:47.6893784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014028.xml
2022-11-23T01:48:47.6894157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6894331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6894710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6894896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6894916Z 
2022-11-23T01:48:47.6895023Z Running tests...
2022-11-23T01:48:47.6895283Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6895579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6895837Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6895858Z 
2022-11-23T01:48:47.6896122Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6896234Z Ran 1 test in 0.003s
2022-11-23T01:48:47.6896253Z 
2022-11-23T01:48:47.6896361Z OK (skipped=1)
2022-11-23T01:48:47.6896380Z 
2022-11-23T01:48:47.6896503Z Generating XML reports...
2022-11-23T01:48:47.6897004Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014031.xml
2022-11-23T01:48:47.6897380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6897553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6898124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6898323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6898344Z 
2022-11-23T01:48:47.6898450Z Running tests...
2022-11-23T01:48:47.6898721Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6899034Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6899283Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6899498Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14974
2022-11-23T01:48:47.6899714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14975
2022-11-23T01:48:47.6900099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6900274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6900650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6900838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6901289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6901461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6901837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6902024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6902264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6902488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6902886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6903280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6903513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6903736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6904072Z STAGE:2022-11-23 01:40:39 14974:14974 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6904394Z STAGE:2022-11-23 01:40:39 14975:14975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6904727Z STAGE:2022-11-23 01:40:40 14974:14974 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6905073Z STAGE:2022-11-23 01:40:40 14974:14974 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6905386Z STAGE:2022-11-23 01:40:40 14975:14975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6905728Z STAGE:2022-11-23 01:40:40 14975:14975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6906053Z STAGE:2022-11-23 01:40:40 14974:14974 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6906369Z STAGE:2022-11-23 01:40:40 14975:14975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.6906755Z STAGE:2022-11-23 01:40:40 14975:14975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6907109Z STAGE:2022-11-23 01:40:40 14975:14975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6907439Z STAGE:2022-11-23 01:40:40 14974:14974 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.6907780Z STAGE:2022-11-23 01:40:40 14974:14974 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.6907866Z ok (7.840s)
2022-11-23T01:48:47.6907903Z 
2022-11-23T01:48:47.6908147Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6908265Z Ran 1 test in 7.840s
2022-11-23T01:48:47.6908284Z 
2022-11-23T01:48:47.6908376Z OK
2022-11-23T01:48:47.6908395Z 
2022-11-23T01:48:47.6908518Z Generating XML reports...
2022-11-23T01:48:47.6909025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014033.xml
2022-11-23T01:48:47.6909395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6909569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6909942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6910113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6910133Z 
2022-11-23T01:48:47.6910241Z Running tests...
2022-11-23T01:48:47.6910502Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6910876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6911134Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6911154Z 
2022-11-23T01:48:47.6911415Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6911527Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6911546Z 
2022-11-23T01:48:47.6911654Z OK (skipped=1)
2022-11-23T01:48:47.6911673Z 
2022-11-23T01:48:47.6911795Z Generating XML reports...
2022-11-23T01:48:47.6912223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014044.xml
2022-11-23T01:48:47.6912588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6912761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6913133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6913325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6913344Z 
2022-11-23T01:48:47.6913452Z Running tests...
2022-11-23T01:48:47.6913722Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6914035Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6914271Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.6914309Z 
2022-11-23T01:48:47.6914551Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6914664Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6914684Z 
2022-11-23T01:48:47.6914792Z OK (skipped=1)
2022-11-23T01:48:47.6914811Z 
2022-11-23T01:48:47.6914933Z Generating XML reports...
2022-11-23T01:48:47.6915372Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014046.xml
2022-11-23T01:48:47.6915744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6915963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6916346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6916515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6916553Z 
2022-11-23T01:48:47.6916644Z Running tests...
2022-11-23T01:48:47.6916903Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6917212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6917464Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6917683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15163
2022-11-23T01:48:47.6917898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15164
2022-11-23T01:48:47.6918269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6918425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6918799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6918987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6919351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6919530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6919903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6920141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6920383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6920627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6921009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6921401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6921625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6921852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6921955Z ok (7.137s)
2022-11-23T01:48:47.6921980Z 
2022-11-23T01:48:47.6922245Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6922358Z Ran 1 test in 7.138s
2022-11-23T01:48:47.6922377Z 
2022-11-23T01:48:47.6922472Z OK
2022-11-23T01:48:47.6922490Z 
2022-11-23T01:48:47.6922616Z Generating XML reports...
2022-11-23T01:48:47.6923048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014048.xml
2022-11-23T01:48:47.6923416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6923588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6923960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6924148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6924167Z 
2022-11-23T01:48:47.6924281Z Running tests...
2022-11-23T01:48:47.6924543Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6924856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6925154Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6925910Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T01:48:47.6925949Z 
2022-11-23T01:48:47.6926188Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6926301Z Ran 1 test in 1.642s
2022-11-23T01:48:47.6926321Z 
2022-11-23T01:48:47.6926430Z OK (skipped=1)
2022-11-23T01:48:47.6926449Z 
2022-11-23T01:48:47.6926578Z Generating XML reports...
2022-11-23T01:48:47.6927025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014058.xml
2022-11-23T01:48:47.6927392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6927639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6928014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6928184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6928204Z 
2022-11-23T01:48:47.6928312Z Running tests...
2022-11-23T01:48:47.6928572Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6928882Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6929129Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6929399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15315
2022-11-23T01:48:47.6929611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15316
2022-11-23T01:48:47.6929985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6930141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6930516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6930702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6931066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6931239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6931618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6931805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6932050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6932292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6932671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6933068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6933294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6933529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6933751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6933982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6934421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6934819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6934924Z ok (4.353s)
2022-11-23T01:48:47.6934944Z 
2022-11-23T01:48:47.6935192Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6935306Z Ran 1 test in 4.353s
2022-11-23T01:48:47.6935326Z 
2022-11-23T01:48:47.6935418Z OK
2022-11-23T01:48:47.6935438Z 
2022-11-23T01:48:47.6935563Z Generating XML reports...
2022-11-23T01:48:47.6936010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014102.xml
2022-11-23T01:48:47.6936385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6936558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6936940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6937128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6937148Z 
2022-11-23T01:48:47.6937239Z Running tests...
2022-11-23T01:48:47.6937507Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6937822Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6938295Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6938521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15422
2022-11-23T01:48:47.6938821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15423
2022-11-23T01:48:47.6939201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6939375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6939733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6939920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6940286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6940458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6940832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6941025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6941266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6941510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6941909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6942289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6942516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6942739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6942840Z ok (6.959s)
2022-11-23T01:48:47.6942860Z 
2022-11-23T01:48:47.6943128Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6943241Z Ran 1 test in 6.960s
2022-11-23T01:48:47.6943261Z 
2022-11-23T01:48:47.6943354Z OK
2022-11-23T01:48:47.6943372Z 
2022-11-23T01:48:47.6943495Z Generating XML reports...
2022-11-23T01:48:47.6943994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014109.xml
2022-11-23T01:48:47.6944373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6944548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6944925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6945115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6945134Z 
2022-11-23T01:48:47.6945243Z Running tests...
2022-11-23T01:48:47.6945509Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6945823Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6946063Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6946265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15537
2022-11-23T01:48:47.6946479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15538
2022-11-23T01:48:47.6946845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6947020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6947398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6947587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6948010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6948185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6948543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6948731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6948971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6949211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6949610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6950003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6950233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6950456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6950562Z ok (4.461s)
2022-11-23T01:48:47.6950585Z 
2022-11-23T01:48:47.6950833Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6950950Z Ran 1 test in 4.461s
2022-11-23T01:48:47.6950969Z 
2022-11-23T01:48:47.6951062Z OK
2022-11-23T01:48:47.6951081Z 
2022-11-23T01:48:47.6951205Z Generating XML reports...
2022-11-23T01:48:47.6951651Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014119.xml
2022-11-23T01:48:47.6952018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6952192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6952573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6952742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6952781Z 
2022-11-23T01:48:47.6952919Z Running tests...
2022-11-23T01:48:47.6953193Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6953512Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6953775Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6953989Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15642
2022-11-23T01:48:47.6954204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15643
2022-11-23T01:48:47.6954571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6954749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6955106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6955298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6955660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6955831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6956204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6956390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6956631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6956927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6957306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6957707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6957937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6958171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6958394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6958624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6959017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6959413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6959519Z ok (4.357s)
2022-11-23T01:48:47.6959540Z 
2022-11-23T01:48:47.6959786Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6959899Z Ran 1 test in 4.357s
2022-11-23T01:48:47.6959919Z 
2022-11-23T01:48:47.6960013Z OK
2022-11-23T01:48:47.6960032Z 
2022-11-23T01:48:47.6960156Z Generating XML reports...
2022-11-23T01:48:47.6960601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014126.xml
2022-11-23T01:48:47.6960969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6961143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6961521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6961712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6961733Z 
2022-11-23T01:48:47.6961824Z Running tests...
2022-11-23T01:48:47.6962147Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6962469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6962728Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6962944Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15749
2022-11-23T01:48:47.6963157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15750
2022-11-23T01:48:47.6963526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6963701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6964062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6964250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6964613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6964789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6965161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6965348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6965589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6965829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6966283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6966657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6966889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6967114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6967351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6967585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6967978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6968365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6968474Z ok (4.263s)
2022-11-23T01:48:47.6968494Z 
2022-11-23T01:48:47.6968758Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6968854Z Ran 1 test in 4.263s
2022-11-23T01:48:47.6968877Z 
2022-11-23T01:48:47.6968971Z OK
2022-11-23T01:48:47.6968990Z 
2022-11-23T01:48:47.6969114Z Generating XML reports...
2022-11-23T01:48:47.6969569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014133.xml
2022-11-23T01:48:47.6969939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6970114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6970491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6970687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6970706Z 
2022-11-23T01:48:47.6970816Z Running tests...
2022-11-23T01:48:47.6971061Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6971422Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6971688Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6971905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15856
2022-11-23T01:48:47.6972119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15857
2022-11-23T01:48:47.6972489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6972664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6973041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6973216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6973582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6973755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6974129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6974317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6974557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6974799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6975194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6975640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6975854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6976077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6976330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9ltswch
2022-11-23T01:48:47.6976598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9ltswch/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6976849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrn1yleb
2022-11-23T01:48:47.6977112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrn1yleb/_remote_module_non_scriptable.py
2022-11-23T01:48:47.6977215Z ok (6.379s)
2022-11-23T01:48:47.6977239Z 
2022-11-23T01:48:47.6977505Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6977600Z Ran 1 test in 6.379s
2022-11-23T01:48:47.6977637Z 
2022-11-23T01:48:47.6977711Z OK
2022-11-23T01:48:47.6977731Z 
2022-11-23T01:48:47.6977853Z Generating XML reports...
2022-11-23T01:48:47.6978540Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014139.xml
2022-11-23T01:48:47.6978912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6979085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6979462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6979649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6979669Z 
2022-11-23T01:48:47.6979781Z Running tests...
2022-11-23T01:48:47.6980025Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6980337Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6980647Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-11-23T01:48:47.6980669Z 
2022-11-23T01:48:47.6980938Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6981049Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6981068Z 
2022-11-23T01:48:47.6981177Z OK (skipped=1)
2022-11-23T01:48:47.6981196Z 
2022-11-23T01:48:47.6981319Z Generating XML reports...
2022-11-23T01:48:47.6981767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014148.xml
2022-11-23T01:48:47.6982137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6982299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6982678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6982865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6982889Z 
2022-11-23T01:48:47.6982997Z Running tests...
2022-11-23T01:48:47.6983256Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6983567Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6983803Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T01:48:47.6983822Z 
2022-11-23T01:48:47.6984083Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6984178Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6984197Z 
2022-11-23T01:48:47.6984305Z OK (skipped=1)
2022-11-23T01:48:47.6984388Z 
2022-11-23T01:48:47.6984520Z Generating XML reports...
2022-11-23T01:48:47.6984972Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014151.xml
2022-11-23T01:48:47.6985342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6985515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6985891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6986079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6986099Z 
2022-11-23T01:48:47.6986206Z Running tests...
2022-11-23T01:48:47.6986447Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6986756Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6987023Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T01:48:47.6987043Z 
2022-11-23T01:48:47.6987302Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6987415Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6987438Z 
2022-11-23T01:48:47.6987545Z OK (skipped=1)
2022-11-23T01:48:47.6987563Z 
2022-11-23T01:48:47.6987686Z Generating XML reports...
2022-11-23T01:48:47.6988127Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014153.xml
2022-11-23T01:48:47.6988493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6988651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6989030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6989222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6989242Z 
2022-11-23T01:48:47.6989352Z Running tests...
2022-11-23T01:48:47.6989612Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6989973Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6990234Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T01:48:47.6990254Z 
2022-11-23T01:48:47.6990517Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6990612Z Ran 1 test in 0.002s
2022-11-23T01:48:47.6990649Z 
2022-11-23T01:48:47.6990741Z OK (skipped=1)
2022-11-23T01:48:47.6990760Z 
2022-11-23T01:48:47.6990883Z Generating XML reports...
2022-11-23T01:48:47.6991329Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014155.xml
2022-11-23T01:48:47.6991702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6991874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6992252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6992440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6992460Z 
2022-11-23T01:48:47.6992567Z Running tests...
2022-11-23T01:48:47.6992809Z ----------------------------------------------------------------------
2022-11-23T01:48:47.6993120Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.6993397Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.6993614Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16106
2022-11-23T01:48:47.6993885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16107
2022-11-23T01:48:47.6994256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6994433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6994806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6994974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6995341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.6995516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.6995891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.6996078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.6996325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.6996566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.6996968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6997366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.6997577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.6997810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.6998028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.6998260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.6998658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6999099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.6999353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:47.6999595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:47.7000037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.7000413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.7000643Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:48:47.7000751Z ok (22.395s)
2022-11-23T01:48:47.7000772Z 
2022-11-23T01:48:47.7001039Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7001153Z Ran 1 test in 22.395s
2022-11-23T01:48:47.7001172Z 
2022-11-23T01:48:47.7001267Z OK
2022-11-23T01:48:47.7001287Z 
2022-11-23T01:48:47.7001409Z Generating XML reports...
2022-11-23T01:48:47.7001859Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014158.xml
2022-11-23T01:48:47.7002228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7002382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7002759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7002946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7003016Z 
2022-11-23T01:48:47.7003129Z Running tests...
2022-11-23T01:48:47.7003393Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7003708Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7004004Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7004222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16229
2022-11-23T01:48:47.7004420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16230
2022-11-23T01:48:47.7004790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7004964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7005340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7005532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7005898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7006069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7006447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7006636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7006861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7007105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7007505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7007905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7008182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7008423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.7008645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7008876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.7009273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.7009643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.7009884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:47.7010119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:47.7010511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.7010907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.7011138Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:48:47.7011367Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:48:47.7011470Z ok (22.398s)
2022-11-23T01:48:47.7011490Z 
2022-11-23T01:48:47.7011755Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7011852Z Ran 1 test in 22.398s
2022-11-23T01:48:47.7011921Z 
2022-11-23T01:48:47.7012018Z OK
2022-11-23T01:48:47.7012038Z 
2022-11-23T01:48:47.7012163Z Generating XML reports...
2022-11-23T01:48:47.7012611Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014223.xml
2022-11-23T01:48:47.7012982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7013157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7013533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7013723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7013743Z 
2022-11-23T01:48:47.7013852Z Running tests...
2022-11-23T01:48:47.7014100Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7014412Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7014827Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.7014848Z 
2022-11-23T01:48:47.7015107Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7015219Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7015239Z 
2022-11-23T01:48:47.7015348Z OK (skipped=1)
2022-11-23T01:48:47.7015367Z 
2022-11-23T01:48:47.7015488Z Generating XML reports...
2022-11-23T01:48:47.7015935Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014248.xml
2022-11-23T01:48:47.7016286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7016459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7016835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7017028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7017049Z 
2022-11-23T01:48:47.7017156Z Running tests...
2022-11-23T01:48:47.7017503Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7017825Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7018450Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.7018472Z 
2022-11-23T01:48:47.7018733Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7018828Z Ran 1 test in 0.003s
2022-11-23T01:48:47.7018870Z 
2022-11-23T01:48:47.7018959Z OK (skipped=1)
2022-11-23T01:48:47.7018978Z 
2022-11-23T01:48:47.7019100Z Generating XML reports...
2022-11-23T01:48:47.7019543Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014250.xml
2022-11-23T01:48:47.7019917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7020095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7020473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7020661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7020681Z 
2022-11-23T01:48:47.7020788Z Running tests...
2022-11-23T01:48:47.7021032Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7021341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7021759Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.7021855Z 
2022-11-23T01:48:47.7022132Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7022245Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7022265Z 
2022-11-23T01:48:47.7022372Z OK (skipped=1)
2022-11-23T01:48:47.7022395Z 
2022-11-23T01:48:47.7022520Z Generating XML reports...
2022-11-23T01:48:47.7022966Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014252.xml
2022-11-23T01:48:47.7023333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7023489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7023865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7024052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7024076Z 
2022-11-23T01:48:47.7024184Z Running tests...
2022-11-23T01:48:47.7024446Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7024757Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7025169Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.7025189Z 
2022-11-23T01:48:47.7025450Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7025545Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7025581Z 
2022-11-23T01:48:47.7025670Z OK (skipped=1)
2022-11-23T01:48:47.7025689Z 
2022-11-23T01:48:47.7025813Z Generating XML reports...
2022-11-23T01:48:47.7026257Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014255.xml
2022-11-23T01:48:47.7026625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7026803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7027252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7027450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7027470Z 
2022-11-23T01:48:47.7027579Z Running tests...
2022-11-23T01:48:47.7027826Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7028137Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7028547Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.7028568Z 
2022-11-23T01:48:47.7028823Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7028939Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7028959Z 
2022-11-23T01:48:47.7029065Z OK (skipped=1)
2022-11-23T01:48:47.7029084Z 
2022-11-23T01:48:47.7029207Z Generating XML reports...
2022-11-23T01:48:47.7029652Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014257.xml
2022-11-23T01:48:47.7030149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7030305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7030682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7030869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7030888Z 
2022-11-23T01:48:47.7030997Z Running tests...
2022-11-23T01:48:47.7031258Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7031630Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7031900Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7032123Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16517
2022-11-23T01:48:47.7032336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16518
2022-11-23T01:48:47.7032685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7032855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7033234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7033422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7033781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7033956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7034331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7034516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7034740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7034979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7035380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7035776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7036009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7036233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7036335Z ok (5.768s)
2022-11-23T01:48:47.7036356Z 
2022-11-23T01:48:47.7036670Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7036790Z Ran 1 test in 5.768s
2022-11-23T01:48:47.7036810Z 
2022-11-23T01:48:47.7036884Z OK
2022-11-23T01:48:47.7036903Z 
2022-11-23T01:48:47.7037029Z Generating XML reports...
2022-11-23T01:48:47.7037478Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014300.xml
2022-11-23T01:48:47.7037850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7038025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7038400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7038594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7038614Z 
2022-11-23T01:48:47.7038722Z Running tests...
2022-11-23T01:48:47.7038968Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7039285Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7039555Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7039771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16631
2022-11-23T01:48:47.7039986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16632
2022-11-23T01:48:47.7040352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7040579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7040960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7041152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7041498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7041669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7042038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7042226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7042468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7042712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7043115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7043514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7043739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7043946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7044047Z ok (5.760s)
2022-11-23T01:48:47.7044066Z 
2022-11-23T01:48:47.7044330Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7044443Z Ran 1 test in 5.761s
2022-11-23T01:48:47.7044463Z 
2022-11-23T01:48:47.7044555Z OK
2022-11-23T01:48:47.7044574Z 
2022-11-23T01:48:47.7044702Z Generating XML reports...
2022-11-23T01:48:47.7045149Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014308.xml
2022-11-23T01:48:47.7045525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7045729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7046118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7046304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7046325Z 
2022-11-23T01:48:47.7046434Z Running tests...
2022-11-23T01:48:47.7046696Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7047008Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7047277Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7047498Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16745
2022-11-23T01:48:47.7047713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16746
2022-11-23T01:48:47.7048068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7048243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7048621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7048810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7049175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7049348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7049722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7049962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7050186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7050431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7050831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7051226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7051450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7051675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7051778Z ok (6.950s)
2022-11-23T01:48:47.7051803Z 
2022-11-23T01:48:47.7052068Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7052182Z Ran 1 test in 6.950s
2022-11-23T01:48:47.7052201Z 
2022-11-23T01:48:47.7052276Z OK
2022-11-23T01:48:47.7052295Z 
2022-11-23T01:48:47.7052418Z Generating XML reports...
2022-11-23T01:48:47.7052871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014316.xml
2022-11-23T01:48:47.7053242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7053416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7053791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7053979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7053999Z 
2022-11-23T01:48:47.7054112Z Running tests...
2022-11-23T01:48:47.7054359Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7054675Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7054985Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7055208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16860
2022-11-23T01:48:47.7055420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16861
2022-11-23T01:48:47.7055786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7055960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7056338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7056527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7056877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7057050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7057421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7057608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7057852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7058311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7058715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7059108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7059417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7059624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7059730Z ok (5.951s)
2022-11-23T01:48:47.7059750Z 
2022-11-23T01:48:47.7060016Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7060128Z Ran 1 test in 5.951s
2022-11-23T01:48:47.7060147Z 
2022-11-23T01:48:47.7060239Z OK
2022-11-23T01:48:47.7060258Z 
2022-11-23T01:48:47.7060383Z Generating XML reports...
2022-11-23T01:48:47.7060830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014326.xml
2022-11-23T01:48:47.7061200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7061362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7061741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7061928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7061951Z 
2022-11-23T01:48:47.7062060Z Running tests...
2022-11-23T01:48:47.7062324Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7062635Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7062902Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7063651Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2022-11-23T01:48:47.7063678Z 
2022-11-23T01:48:47.7063940Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7064054Z Ran 1 test in 1.621s
2022-11-23T01:48:47.7064073Z 
2022-11-23T01:48:47.7064164Z OK (skipped=1)
2022-11-23T01:48:47.7064245Z 
2022-11-23T01:48:47.7064379Z Generating XML reports...
2022-11-23T01:48:47.7064831Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014334.xml
2022-11-23T01:48:47.7065198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7065373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7065751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7065939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7065963Z 
2022-11-23T01:48:47.7066071Z Running tests...
2022-11-23T01:48:47.7066318Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7066637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7066887Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:47.7066908Z 
2022-11-23T01:48:47.7067169Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7067282Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7067301Z 
2022-11-23T01:48:47.7067408Z OK (skipped=1)
2022-11-23T01:48:47.7067428Z 
2022-11-23T01:48:47.7067550Z Generating XML reports...
2022-11-23T01:48:47.7067998Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014338.xml
2022-11-23T01:48:47.7068368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7068578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7068959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7069150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7069170Z 
2022-11-23T01:48:47.7069277Z Running tests...
2022-11-23T01:48:47.7069542Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7069854Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7070123Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:47.7070143Z 
2022-11-23T01:48:47.7070402Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7070516Z Ran 1 test in 0.003s
2022-11-23T01:48:47.7070539Z 
2022-11-23T01:48:47.7070633Z OK (skipped=1)
2022-11-23T01:48:47.7070652Z 
2022-11-23T01:48:47.7070776Z Generating XML reports...
2022-11-23T01:48:47.7071230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014341.xml
2022-11-23T01:48:47.7071599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7071772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7072154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7072344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7072364Z 
2022-11-23T01:48:47.7072472Z Running tests...
2022-11-23T01:48:47.7072715Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7073035Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7073344Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:47.7073365Z 
2022-11-23T01:48:47.7073678Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7073796Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7073817Z 
2022-11-23T01:48:47.7073924Z OK (skipped=1)
2022-11-23T01:48:47.7073944Z 
2022-11-23T01:48:47.7074068Z Generating XML reports...
2022-11-23T01:48:47.7074519Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014343.xml
2022-11-23T01:48:47.7074888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7075045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7075430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7075619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7075639Z 
2022-11-23T01:48:47.7075745Z Running tests...
2022-11-23T01:48:47.7076012Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7076325Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7076623Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7076842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17107
2022-11-23T01:48:47.7077057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17108
2022-11-23T01:48:47.7077413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7077641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7078025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7078214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7078578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7078748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7079119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7079304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7079529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7079767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7080175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7080572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7080801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7081025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7081127Z ok (4.277s)
2022-11-23T01:48:47.7081147Z 
2022-11-23T01:48:47.7081410Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7081522Z Ran 1 test in 4.277s
2022-11-23T01:48:47.7081542Z 
2022-11-23T01:48:47.7081616Z OK
2022-11-23T01:48:47.7081635Z 
2022-11-23T01:48:47.7081759Z Generating XML reports...
2022-11-23T01:48:47.7082214Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014345.xml
2022-11-23T01:48:47.7082582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7082801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7083185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7083372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7083391Z 
2022-11-23T01:48:47.7083499Z Running tests...
2022-11-23T01:48:47.7083760Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7084054Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7084346Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7084567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17212
2022-11-23T01:48:47.7084782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17213
2022-11-23T01:48:47.7085158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7085332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7085706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7085897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7086239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7086411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7086837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7087025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7087271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7087511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7087908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7088304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7088533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7088739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7088846Z ok (4.251s)
2022-11-23T01:48:47.7088867Z 
2022-11-23T01:48:47.7089130Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7089244Z Ran 1 test in 4.251s
2022-11-23T01:48:47.7089264Z 
2022-11-23T01:48:47.7089356Z OK
2022-11-23T01:48:47.7089375Z 
2022-11-23T01:48:47.7089502Z Generating XML reports...
2022-11-23T01:48:47.7089949Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014352.xml
2022-11-23T01:48:47.7090322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7090497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7090858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7091046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7091069Z 
2022-11-23T01:48:47.7091177Z Running tests...
2022-11-23T01:48:47.7091434Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7091747Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7092079Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:47.7092100Z 
2022-11-23T01:48:47.7092368Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7092479Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7092498Z 
2022-11-23T01:48:47.7092588Z OK (skipped=1)
2022-11-23T01:48:47.7092626Z 
2022-11-23T01:48:47.7092731Z Generating XML reports...
2022-11-23T01:48:47.7093177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014359.xml
2022-11-23T01:48:47.7093543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7093720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7094093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7094285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7094305Z 
2022-11-23T01:48:47.7094414Z Running tests...
2022-11-23T01:48:47.7094673Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7094964Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7095257Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:47.7095277Z 
2022-11-23T01:48:47.7095532Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7095699Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7095719Z 
2022-11-23T01:48:47.7095825Z OK (skipped=1)
2022-11-23T01:48:47.7095845Z 
2022-11-23T01:48:47.7095969Z Generating XML reports...
2022-11-23T01:48:47.7096423Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014402.xml
2022-11-23T01:48:47.7096789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7096962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7097320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7097509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7097529Z 
2022-11-23T01:48:47.7097636Z Running tests...
2022-11-23T01:48:47.7098112Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7098449Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7098725Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7099470Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.657s)
2022-11-23T01:48:47.7099491Z 
2022-11-23T01:48:47.7099750Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7099889Z Ran 1 test in 1.657s
2022-11-23T01:48:47.7099911Z 
2022-11-23T01:48:47.7100002Z OK (skipped=1)
2022-11-23T01:48:47.7100040Z 
2022-11-23T01:48:47.7100145Z Generating XML reports...
2022-11-23T01:48:47.7100588Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014404.xml
2022-11-23T01:48:47.7100962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7101135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7101587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7101784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7101803Z 
2022-11-23T01:48:47.7101911Z Running tests...
2022-11-23T01:48:47.7102178Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7102472Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7102747Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7102971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17417
2022-11-23T01:48:47.7103185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17418
2022-11-23T01:48:47.7103555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7103729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7104106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7104295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7104654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7104807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7105174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7105432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7105677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7105922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7106320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7106717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7106944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7107149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7107403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpva35kjim
2022-11-23T01:48:47.7107675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpva35kjim/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7107923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl443rsq
2022-11-23T01:48:47.7108187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl443rsq/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7108290Z ok (7.358s)
2022-11-23T01:48:47.7108310Z 
2022-11-23T01:48:47.7108577Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7108689Z Ran 1 test in 7.358s
2022-11-23T01:48:47.7108709Z 
2022-11-23T01:48:47.7108799Z OK
2022-11-23T01:48:47.7108818Z 
2022-11-23T01:48:47.7108923Z Generating XML reports...
2022-11-23T01:48:47.7109370Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014408.xml
2022-11-23T01:48:47.7109738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7109916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7110290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7110526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7110547Z 
2022-11-23T01:48:47.7110651Z Running tests...
2022-11-23T01:48:47.7122385Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7122806Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7123087Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7123313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17536
2022-11-23T01:48:47.7123530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17537
2022-11-23T01:48:47.7123922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7124100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7124465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7124663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7125028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7125202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7125581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7125771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7126144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7126389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7126776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7127172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7127402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7127629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7127735Z ok (6.838s)
2022-11-23T01:48:47.7127758Z 
2022-11-23T01:48:47.7128026Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7128138Z Ran 1 test in 6.838s
2022-11-23T01:48:47.7128163Z 
2022-11-23T01:48:47.7128257Z OK
2022-11-23T01:48:47.7128275Z 
2022-11-23T01:48:47.7128400Z Generating XML reports...
2022-11-23T01:48:47.7128837Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014418.xml
2022-11-23T01:48:47.7129211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7129392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7129772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7129960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7129979Z 
2022-11-23T01:48:47.7130090Z Running tests...
2022-11-23T01:48:47.7130351Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7130664Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7130956Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7131159Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17651
2022-11-23T01:48:47.7131431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17652
2022-11-23T01:48:47.7131813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7131990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7132366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7132555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7132913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7133093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7133447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7133638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7133881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7134122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7134521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7134917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7135145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7135432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7135536Z ok (7.001s)
2022-11-23T01:48:47.7135557Z 
2022-11-23T01:48:47.7135808Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7135922Z Ran 1 test in 7.001s
2022-11-23T01:48:47.7135942Z 
2022-11-23T01:48:47.7136497Z OK
2022-11-23T01:48:47.7136518Z 
2022-11-23T01:48:47.7136648Z Generating XML reports...
2022-11-23T01:48:47.7137107Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014428.xml
2022-11-23T01:48:47.7137476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7137653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7138336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7138541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7138563Z 
2022-11-23T01:48:47.7138654Z Running tests...
2022-11-23T01:48:47.7138931Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7139245Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7139525Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7140274Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2022-11-23T01:48:47.7140295Z 
2022-11-23T01:48:47.7140558Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7140676Z Ran 1 test in 1.633s
2022-11-23T01:48:47.7140696Z 
2022-11-23T01:48:47.7140808Z OK (skipped=1)
2022-11-23T01:48:47.7140827Z 
2022-11-23T01:48:47.7140952Z Generating XML reports...
2022-11-23T01:48:47.7141487Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014437.xml
2022-11-23T01:48:47.7141878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7142052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7142432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7142624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7142644Z 
2022-11-23T01:48:47.7142754Z Running tests...
2022-11-23T01:48:47.7143019Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7143339Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7143634Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7144380Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.607s)
2022-11-23T01:48:47.7144402Z 
2022-11-23T01:48:47.7144645Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7144760Z Ran 1 test in 1.607s
2022-11-23T01:48:47.7144779Z 
2022-11-23T01:48:47.7144889Z OK (skipped=1)
2022-11-23T01:48:47.7144908Z 
2022-11-23T01:48:47.7145033Z Generating XML reports...
2022-11-23T01:48:47.7145479Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014441.xml
2022-11-23T01:48:47.7145929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7146108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7146485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7146677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7146697Z 
2022-11-23T01:48:47.7146787Z Running tests...
2022-11-23T01:48:47.7147051Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7147367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7147677Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7147902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17834
2022-11-23T01:48:47.7148119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17835
2022-11-23T01:48:47.7148494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7148673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7149033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7149223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7149589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7149763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7150146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7150335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7150632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7150880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7151283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7151656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7151888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7152112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7152268Z skip: Need at least 4 CUDA devices (4.352s)
2022-11-23T01:48:47.7152288Z 
2022-11-23T01:48:47.7152552Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7152670Z Ran 1 test in 4.352s
2022-11-23T01:48:47.7152689Z 
2022-11-23T01:48:47.7152802Z OK (skipped=1)
2022-11-23T01:48:47.7152822Z 
2022-11-23T01:48:47.7152947Z Generating XML reports...
2022-11-23T01:48:47.7153399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014445.xml
2022-11-23T01:48:47.7153751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7153926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7154304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7154491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7154560Z 
2022-11-23T01:48:47.7154673Z Running tests...
2022-11-23T01:48:47.7154941Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7155261Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7155585Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7155783Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17939
2022-11-23T01:48:47.7156001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17940
2022-11-23T01:48:47.7156370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7156545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7156921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7157117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7157489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7157665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7158037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7158206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7158450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7158691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7159092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7159493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7159773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7160005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7160157Z skip: Need at least 4 CUDA devices (4.254s)
2022-11-23T01:48:47.7160178Z 
2022-11-23T01:48:47.7160443Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7160538Z Ran 1 test in 4.254s
2022-11-23T01:48:47.7160558Z 
2022-11-23T01:48:47.7160668Z OK (skipped=1)
2022-11-23T01:48:47.7160687Z 
2022-11-23T01:48:47.7160813Z Generating XML reports...
2022-11-23T01:48:47.7161265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014452.xml
2022-11-23T01:48:47.7161638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7161817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7162193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7162384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7162404Z 
2022-11-23T01:48:47.7162514Z Running tests...
2022-11-23T01:48:47.7162761Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7163074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7163358Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7164101Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.638s)
2022-11-23T01:48:47.7164170Z 
2022-11-23T01:48:47.7164440Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7164556Z Ran 1 test in 1.638s
2022-11-23T01:48:47.7164576Z 
2022-11-23T01:48:47.7164685Z OK (skipped=1)
2022-11-23T01:48:47.7164704Z 
2022-11-23T01:48:47.7164832Z Generating XML reports...
2022-11-23T01:48:47.7165278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014459.xml
2022-11-23T01:48:47.7165626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7165804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7166185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7166378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7166398Z 
2022-11-23T01:48:47.7166508Z Running tests...
2022-11-23T01:48:47.7166773Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7167087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7167353Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7167372Z 
2022-11-23T01:48:47.7167631Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7167725Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7167745Z 
2022-11-23T01:48:47.7167856Z OK (skipped=1)
2022-11-23T01:48:47.7167876Z 
2022-11-23T01:48:47.7168001Z Generating XML reports...
2022-11-23T01:48:47.7168446Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014503.xml
2022-11-23T01:48:47.7168818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7169058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7169444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7169633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7169652Z 
2022-11-23T01:48:47.7169763Z Running tests...
2022-11-23T01:48:47.7170004Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7170316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7170584Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7170607Z 
2022-11-23T01:48:47.7170865Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7170979Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7170998Z 
2022-11-23T01:48:47.7171107Z OK (skipped=1)
2022-11-23T01:48:47.7171126Z 
2022-11-23T01:48:47.7171254Z Generating XML reports...
2022-11-23T01:48:47.7171700Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014506.xml
2022-11-23T01:48:47.7172045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7172220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7172599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7172790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7172894Z 
2022-11-23T01:48:47.7173007Z Running tests...
2022-11-23T01:48:47.7173270Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7173586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7173863Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7173883Z 
2022-11-23T01:48:47.7174143Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7174235Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7174255Z 
2022-11-23T01:48:47.7174364Z OK (skipped=1)
2022-11-23T01:48:47.7174383Z 
2022-11-23T01:48:47.7174508Z Generating XML reports...
2022-11-23T01:48:47.7174951Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014508.xml
2022-11-23T01:48:47.7175314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7175493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7175872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7176066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7176086Z 
2022-11-23T01:48:47.7176196Z Running tests...
2022-11-23T01:48:47.7176442Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7176755Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7177021Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7177040Z 
2022-11-23T01:48:47.7177300Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7177414Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7177437Z 
2022-11-23T01:48:47.7177547Z OK (skipped=1)
2022-11-23T01:48:47.7177566Z 
2022-11-23T01:48:47.7177691Z Generating XML reports...
2022-11-23T01:48:47.7178321Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014511.xml
2022-11-23T01:48:47.7178776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7178940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7179316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7179506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7179526Z 
2022-11-23T01:48:47.7179637Z Running tests...
2022-11-23T01:48:47.7179899Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7180212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7180481Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7180501Z 
2022-11-23T01:48:47.7180766Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7180859Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7180899Z 
2022-11-23T01:48:47.7180988Z OK (skipped=1)
2022-11-23T01:48:47.7181007Z 
2022-11-23T01:48:47.7181132Z Generating XML reports...
2022-11-23T01:48:47.7181581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014513.xml
2022-11-23T01:48:47.7181950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7182127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7182503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7182762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7182782Z 
2022-11-23T01:48:47.7182893Z Running tests...
2022-11-23T01:48:47.7183141Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7183456Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7183717Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7183736Z 
2022-11-23T01:48:47.7183995Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7184111Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7184130Z 
2022-11-23T01:48:47.7184240Z OK (skipped=1)
2022-11-23T01:48:47.7184259Z 
2022-11-23T01:48:47.7184383Z Generating XML reports...
2022-11-23T01:48:47.7184832Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014515.xml
2022-11-23T01:48:47.7185205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7185361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7185740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7185931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7185950Z 
2022-11-23T01:48:47.7186059Z Running tests...
2022-11-23T01:48:47.7186323Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7186637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7186903Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7186927Z 
2022-11-23T01:48:47.7187185Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7187278Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7187317Z 
2022-11-23T01:48:47.7187407Z OK (skipped=1)
2022-11-23T01:48:47.7187427Z 
2022-11-23T01:48:47.7187598Z Generating XML reports...
2022-11-23T01:48:47.7188052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014518.xml
2022-11-23T01:48:47.7188415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7188589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7188964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7189151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7189171Z 
2022-11-23T01:48:47.7189286Z Running tests...
2022-11-23T01:48:47.7189529Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7189841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7190103Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7190123Z 
2022-11-23T01:48:47.7190384Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7190498Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7190517Z 
2022-11-23T01:48:47.7190626Z OK (skipped=1)
2022-11-23T01:48:47.7190645Z 
2022-11-23T01:48:47.7190770Z Generating XML reports...
2022-11-23T01:48:47.7191217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014520.xml
2022-11-23T01:48:47.7191586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7191799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7192178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7192372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7192392Z 
2022-11-23T01:48:47.7192502Z Running tests...
2022-11-23T01:48:47.7192765Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7193079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7193332Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7193352Z 
2022-11-23T01:48:47.7193615Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7193710Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7193749Z 
2022-11-23T01:48:47.7193839Z OK (skipped=1)
2022-11-23T01:48:47.7193862Z 
2022-11-23T01:48:47.7193987Z Generating XML reports...
2022-11-23T01:48:47.7194439Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014522.xml
2022-11-23T01:48:47.7194810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7194987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7195367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7195561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7195580Z 
2022-11-23T01:48:47.7195690Z Running tests...
2022-11-23T01:48:47.7195931Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7196243Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7196499Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7196519Z 
2022-11-23T01:48:47.7196777Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7196939Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7196960Z 
2022-11-23T01:48:47.7197142Z OK (skipped=1)
2022-11-23T01:48:47.7197161Z 
2022-11-23T01:48:47.7197284Z Generating XML reports...
2022-11-23T01:48:47.7197737Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014525.xml
2022-11-23T01:48:47.7198107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7198263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7198641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7198834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7198854Z 
2022-11-23T01:48:47.7198964Z Running tests...
2022-11-23T01:48:47.7199224Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7199542Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7199801Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7200064Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18408
2022-11-23T01:48:47.7200262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18409
2022-11-23T01:48:47.7200632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7200805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7201181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7201423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7201789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7201964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7202339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7202526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7202751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7202992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7203393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7203796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7204028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7204259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7204597Z STAGE:2022-11-23 01:45:33 18409:18409 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7205365Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:48:47.7205482Z   warnings.warn(
2022-11-23T01:48:47.7205795Z STAGE:2022-11-23 01:45:33 18408:18408 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7206597Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:48:47.7206718Z   warnings.warn(
2022-11-23T01:48:47.7207053Z STAGE:2022-11-23 01:45:33 18408:18408 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7207387Z STAGE:2022-11-23 01:45:33 18409:18409 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7207735Z STAGE:2022-11-23 01:45:33 18408:18408 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7208084Z STAGE:2022-11-23 01:45:33 18409:18409 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7208413Z STAGE:2022-11-23 01:45:33 18409:18409 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7208748Z STAGE:2022-11-23 01:45:33 18409:18409 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7209149Z STAGE:2022-11-23 01:45:33 18409:18409 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7209458Z STAGE:2022-11-23 01:45:33 18408:18408 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7209789Z STAGE:2022-11-23 01:45:33 18408:18408 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7210132Z STAGE:2022-11-23 01:45:33 18408:18408 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7210238Z ok (6.482s)
2022-11-23T01:48:47.7210258Z 
2022-11-23T01:48:47.7210521Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7210636Z Ran 1 test in 6.482s
2022-11-23T01:48:47.7210655Z 
2022-11-23T01:48:47.7210806Z OK
2022-11-23T01:48:47.7210826Z 
2022-11-23T01:48:47.7210955Z Generating XML reports...
2022-11-23T01:48:47.7211394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014527.xml
2022-11-23T01:48:47.7211769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7211947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7212326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7212516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7212536Z 
2022-11-23T01:48:47.7212646Z Running tests...
2022-11-23T01:48:47.7212908Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7213227Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7213495Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7213516Z 
2022-11-23T01:48:47.7213755Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7213873Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7213893Z 
2022-11-23T01:48:47.7214003Z OK (skipped=1)
2022-11-23T01:48:47.7214022Z 
2022-11-23T01:48:47.7214148Z Generating XML reports...
2022-11-23T01:48:47.7214601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014536.xml
2022-11-23T01:48:47.7214971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7215148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7215526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7215718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7215738Z 
2022-11-23T01:48:47.7215828Z Running tests...
2022-11-23T01:48:47.7216094Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7216456Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7216731Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7216952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18559
2022-11-23T01:48:47.7217169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18560
2022-11-23T01:48:47.7217536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7217713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7218293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7218487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7218858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7219035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7219411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7219599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7219843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7220091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7220492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7220962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7221193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7221421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7221758Z STAGE:2022-11-23 01:45:44 18560:18560 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7222082Z STAGE:2022-11-23 01:45:44 18559:18559 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7222418Z STAGE:2022-11-23 01:45:46 18559:18559 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7222752Z STAGE:2022-11-23 01:45:46 18560:18560 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7223108Z STAGE:2022-11-23 01:45:46 18559:18559 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7223451Z STAGE:2022-11-23 01:45:46 18560:18560 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7223761Z STAGE:2022-11-23 01:45:46 18560:18560 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7224091Z STAGE:2022-11-23 01:45:46 18560:18560 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7224439Z STAGE:2022-11-23 01:45:46 18560:18560 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7224762Z STAGE:2022-11-23 01:45:46 18559:18559 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7225092Z STAGE:2022-11-23 01:45:46 18559:18559 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7225433Z STAGE:2022-11-23 01:45:46 18559:18559 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7225544Z ok (7.898s)
2022-11-23T01:48:47.7225566Z 
2022-11-23T01:48:47.7225833Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7225928Z Ran 1 test in 7.898s
2022-11-23T01:48:47.7225966Z 
2022-11-23T01:48:47.7226039Z OK
2022-11-23T01:48:47.7226130Z 
2022-11-23T01:48:47.7226267Z Generating XML reports...
2022-11-23T01:48:47.7226721Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014539.xml
2022-11-23T01:48:47.7227093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7227271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7227651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7227843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7227867Z 
2022-11-23T01:48:47.7227980Z Running tests...
2022-11-23T01:48:47.7228230Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7228544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7228809Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7229027Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18678
2022-11-23T01:48:47.7229245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18679
2022-11-23T01:48:47.7229617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7229793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7230169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7230399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7230765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7230942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7231314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7231503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7231750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7231991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7232390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7232794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7233003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7233236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7233341Z ok (7.054s)
2022-11-23T01:48:47.7233361Z 
2022-11-23T01:48:47.7233626Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7233741Z Ran 1 test in 7.054s
2022-11-23T01:48:47.7233761Z 
2022-11-23T01:48:47.7233854Z OK
2022-11-23T01:48:47.7233873Z 
2022-11-23T01:48:47.7234000Z Generating XML reports...
2022-11-23T01:48:47.7234451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014549.xml
2022-11-23T01:48:47.7234821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7234983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7235356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7235614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7235636Z 
2022-11-23T01:48:47.7235749Z Running tests...
2022-11-23T01:48:47.7236017Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7236330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7236585Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7236605Z 
2022-11-23T01:48:47.7236864Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7236959Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7237001Z 
2022-11-23T01:48:47.7237091Z OK (skipped=1)
2022-11-23T01:48:47.7237110Z 
2022-11-23T01:48:47.7237235Z Generating XML reports...
2022-11-23T01:48:47.7237683Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014559.xml
2022-11-23T01:48:47.7238057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7238231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7238609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7238801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7238820Z 
2022-11-23T01:48:47.7238932Z Running tests...
2022-11-23T01:48:47.7239178Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7239491Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7239796Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7240014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18826
2022-11-23T01:48:47.7240234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18827
2022-11-23T01:48:47.7240605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7240780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7241145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7241300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7241676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7241869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7242241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7242434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7242680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7242922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7243322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7243719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7243928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7244159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7244493Z STAGE:2022-11-23 01:46:07 18826:18826 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7244865Z STAGE:2022-11-23 01:46:07 18827:18827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7245210Z STAGE:2022-11-23 01:46:07 18827:18827 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7245538Z STAGE:2022-11-23 01:46:07 18826:18826 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7245884Z STAGE:2022-11-23 01:46:07 18827:18827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7246230Z STAGE:2022-11-23 01:46:07 18826:18826 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7246557Z STAGE:2022-11-23 01:46:07 18827:18827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7246879Z STAGE:2022-11-23 01:46:07 18827:18827 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7247224Z STAGE:2022-11-23 01:46:07 18827:18827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7247547Z STAGE:2022-11-23 01:46:07 18826:18826 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7247878Z STAGE:2022-11-23 01:46:07 18826:18826 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7248220Z STAGE:2022-11-23 01:46:07 18826:18826 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7248325Z ok (6.321s)
2022-11-23T01:48:47.7248345Z 
2022-11-23T01:48:47.7248612Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7248727Z Ran 1 test in 6.322s
2022-11-23T01:48:47.7248746Z 
2022-11-23T01:48:47.7248840Z OK
2022-11-23T01:48:47.7248859Z 
2022-11-23T01:48:47.7249022Z Generating XML reports...
2022-11-23T01:48:47.7249474Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014601.xml
2022-11-23T01:48:47.7249849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7250027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7250406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7250598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7250618Z 
2022-11-23T01:48:47.7250730Z Running tests...
2022-11-23T01:48:47.7250994Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7251288Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7251549Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7251769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18944
2022-11-23T01:48:47.7251985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18945
2022-11-23T01:48:47.7252360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7252534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7252909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7253101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7253462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7253615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7253994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7254183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7254476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7254725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7255122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7255519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7255749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7255975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7256295Z STAGE:2022-11-23 01:46:16 18944:18944 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7256619Z STAGE:2022-11-23 01:46:16 18945:18945 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7256962Z STAGE:2022-11-23 01:46:16 18945:18945 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7257299Z STAGE:2022-11-23 01:46:16 18944:18944 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7257645Z STAGE:2022-11-23 01:46:16 18945:18945 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7258231Z STAGE:2022-11-23 01:46:16 18944:18944 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7258568Z STAGE:2022-11-23 01:46:16 18945:18945 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7258902Z STAGE:2022-11-23 01:46:16 18945:18945 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7259320Z STAGE:2022-11-23 01:46:16 18945:18945 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7259647Z STAGE:2022-11-23 01:46:16 18944:18944 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7259975Z STAGE:2022-11-23 01:46:16 18944:18944 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7260314Z STAGE:2022-11-23 01:46:16 18944:18944 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7260419Z ok (6.458s)
2022-11-23T01:48:47.7260439Z 
2022-11-23T01:48:47.7260702Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7260816Z Ran 1 test in 6.458s
2022-11-23T01:48:47.7260836Z 
2022-11-23T01:48:47.7260930Z OK
2022-11-23T01:48:47.7260948Z 
2022-11-23T01:48:47.7261077Z Generating XML reports...
2022-11-23T01:48:47.7261505Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014610.xml
2022-11-23T01:48:47.7261880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7262061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7262440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7262630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7262650Z 
2022-11-23T01:48:47.7262760Z Running tests...
2022-11-23T01:48:47.7263021Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7263339Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7263602Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7263626Z 
2022-11-23T01:48:47.7263864Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7263978Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7263998Z 
2022-11-23T01:48:47.7264111Z OK (skipped=1)
2022-11-23T01:48:47.7264130Z 
2022-11-23T01:48:47.7264318Z Generating XML reports...
2022-11-23T01:48:47.7264775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014619.xml
2022-11-23T01:48:47.7265146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7265322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7265698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7265869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7265908Z 
2022-11-23T01:48:47.7266003Z Running tests...
2022-11-23T01:48:47.7266264Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7266578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7266834Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7266854Z 
2022-11-23T01:48:47.7267116Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7267229Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7267248Z 
2022-11-23T01:48:47.7267357Z OK (skipped=1)
2022-11-23T01:48:47.7267376Z 
2022-11-23T01:48:47.7267501Z Generating XML reports...
2022-11-23T01:48:47.7267926Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014621.xml
2022-11-23T01:48:47.7268295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7268524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7268907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7269100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7269120Z 
2022-11-23T01:48:47.7269231Z Running tests...
2022-11-23T01:48:47.7269495Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7269809Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7270068Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7270088Z 
2022-11-23T01:48:47.7270326Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7270441Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7270460Z 
2022-11-23T01:48:47.7270573Z OK (skipped=1)
2022-11-23T01:48:47.7270592Z 
2022-11-23T01:48:47.7270718Z Generating XML reports...
2022-11-23T01:48:47.7271164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014624.xml
2022-11-23T01:48:47.7271537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7271714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7272093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7272264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7272303Z 
2022-11-23T01:48:47.7272394Z Running tests...
2022-11-23T01:48:47.7272655Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7272969Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7273235Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7273254Z 
2022-11-23T01:48:47.7273514Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7273676Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7273697Z 
2022-11-23T01:48:47.7273810Z OK (skipped=1)
2022-11-23T01:48:47.7273829Z 
2022-11-23T01:48:47.7273953Z Generating XML reports...
2022-11-23T01:48:47.7274384Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014626.xml
2022-11-23T01:48:47.7274752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7274930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7275308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7275501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7275520Z 
2022-11-23T01:48:47.7275631Z Running tests...
2022-11-23T01:48:47.7275895Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7276211Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7276465Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7276665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19194
2022-11-23T01:48:47.7276884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19195
2022-11-23T01:48:47.7277253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7277429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7277860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7278051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7278419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7278598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7278948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7279138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7279382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7279625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7280025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7280431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7280663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7280886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7281223Z STAGE:2022-11-23 01:46:34 19194:19194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7281527Z STAGE:2022-11-23 01:46:34 19195:19195 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7281863Z STAGE:2022-11-23 01:46:36 19194:19194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7282212Z STAGE:2022-11-23 01:46:36 19194:19194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7282551Z STAGE:2022-11-23 01:46:36 19195:19195 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7282894Z STAGE:2022-11-23 01:46:36 19195:19195 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7283268Z STAGE:2022-11-23 01:46:36 19194:19194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7283594Z STAGE:2022-11-23 01:46:36 19195:19195 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7283927Z STAGE:2022-11-23 01:46:36 19195:19195 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7284274Z STAGE:2022-11-23 01:46:36 19195:19195 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7284584Z STAGE:2022-11-23 01:46:36 19194:19194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7284929Z STAGE:2022-11-23 01:46:36 19194:19194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7285039Z ok (7.867s)
2022-11-23T01:48:47.7285059Z 
2022-11-23T01:48:47.7285323Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7285437Z Ran 1 test in 7.867s
2022-11-23T01:48:47.7285457Z 
2022-11-23T01:48:47.7285550Z OK
2022-11-23T01:48:47.7285572Z 
2022-11-23T01:48:47.7285698Z Generating XML reports...
2022-11-23T01:48:47.7286147Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014629.xml
2022-11-23T01:48:47.7286520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7286676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7287053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7287246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7287314Z 
2022-11-23T01:48:47.7287427Z Running tests...
2022-11-23T01:48:47.7287693Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7288012Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7288279Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7288500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19317
2022-11-23T01:48:47.7288699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19318
2022-11-23T01:48:47.7289073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7289340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7289838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7290016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7290393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7290588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7290964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7291154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7291380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7291722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7292128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7292527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7292757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7293043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7293388Z STAGE:2022-11-23 01:46:45 19318:19318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7293713Z STAGE:2022-11-23 01:46:45 19317:19317 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7294025Z STAGE:2022-11-23 01:46:46 19317:19317 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7294376Z STAGE:2022-11-23 01:46:46 19317:19317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7294711Z STAGE:2022-11-23 01:46:46 19318:19318 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7295061Z STAGE:2022-11-23 01:46:46 19318:19318 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7295388Z STAGE:2022-11-23 01:46:46 19318:19318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7295711Z STAGE:2022-11-23 01:46:46 19317:19317 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7296044Z STAGE:2022-11-23 01:46:46 19318:19318 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7296389Z STAGE:2022-11-23 01:46:46 19318:19318 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7296720Z STAGE:2022-11-23 01:46:46 19317:19317 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7297046Z STAGE:2022-11-23 01:46:46 19317:19317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7297152Z ok (7.770s)
2022-11-23T01:48:47.7297173Z 
2022-11-23T01:48:47.7297493Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7297609Z Ran 1 test in 7.770s
2022-11-23T01:48:47.7297629Z 
2022-11-23T01:48:47.7297722Z OK
2022-11-23T01:48:47.7297741Z 
2022-11-23T01:48:47.7297866Z Generating XML reports...
2022-11-23T01:48:47.7298498Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014639.xml
2022-11-23T01:48:47.7298869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7299045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7299400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7299600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7299619Z 
2022-11-23T01:48:47.7299731Z Running tests...
2022-11-23T01:48:47.7300032Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7300348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7300618Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7300638Z 
2022-11-23T01:48:47.7300899Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7301012Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7301031Z 
2022-11-23T01:48:47.7301141Z OK (skipped=1)
2022-11-23T01:48:47.7301161Z 
2022-11-23T01:48:47.7301267Z Generating XML reports...
2022-11-23T01:48:47.7301711Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014649.xml
2022-11-23T01:48:47.7302081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7302260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7302632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7302823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7302932Z 
2022-11-23T01:48:47.7303050Z Running tests...
2022-11-23T01:48:47.7303314Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7303610Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7303868Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:48:47.7303888Z 
2022-11-23T01:48:47.7304152Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7304268Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7304288Z 
2022-11-23T01:48:47.7304396Z OK (skipped=1)
2022-11-23T01:48:47.7304419Z 
2022-11-23T01:48:47.7304545Z Generating XML reports...
2022-11-23T01:48:47.7304992Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014652.xml
2022-11-23T01:48:47.7305366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7305545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7305903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7306092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7306112Z 
2022-11-23T01:48:47.7306221Z Running tests...
2022-11-23T01:48:47.7306485Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7306799Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7307257Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:48:47.7307277Z 
2022-11-23T01:48:47.7307536Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7307651Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7307674Z 
2022-11-23T01:48:47.7307784Z OK (skipped=1)
2022-11-23T01:48:47.7307803Z 
2022-11-23T01:48:47.7307908Z Generating XML reports...
2022-11-23T01:48:47.7308354Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014654.xml
2022-11-23T01:48:47.7308723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7308901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7309277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7309472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7309492Z 
2022-11-23T01:48:47.7309603Z Running tests...
2022-11-23T01:48:47.7309868Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7310165Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7310430Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:48:47.7310451Z 
2022-11-23T01:48:47.7310711Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7310825Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7310844Z 
2022-11-23T01:48:47.7310953Z OK (skipped=1)
2022-11-23T01:48:47.7310972Z 
2022-11-23T01:48:47.7311096Z Generating XML reports...
2022-11-23T01:48:47.7311542Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014657.xml
2022-11-23T01:48:47.7311914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7312089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7312493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7312691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7312711Z 
2022-11-23T01:48:47.7312821Z Running tests...
2022-11-23T01:48:47.7313084Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7313397Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7313679Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T01:48:47.7313699Z 
2022-11-23T01:48:47.7313959Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7314078Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7314097Z 
2022-11-23T01:48:47.7314208Z OK (skipped=1)
2022-11-23T01:48:47.7314227Z 
2022-11-23T01:48:47.7314332Z Generating XML reports...
2022-11-23T01:48:47.7314785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014659.xml
2022-11-23T01:48:47.7315151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7315327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7315707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7315896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7315916Z 
2022-11-23T01:48:47.7316026Z Running tests...
2022-11-23T01:48:47.7316290Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7316664Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7316954Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T01:48:47.7316975Z 
2022-11-23T01:48:47.7317235Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7317351Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7317371Z 
2022-11-23T01:48:47.7317479Z OK (skipped=1)
2022-11-23T01:48:47.7317499Z 
2022-11-23T01:48:47.7317624Z Generating XML reports...
2022-11-23T01:48:47.7318070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014701.xml
2022-11-23T01:48:47.7318438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7318618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7318992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7319163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7319183Z 
2022-11-23T01:48:47.7319293Z Running tests...
2022-11-23T01:48:47.7319555Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7319869Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7320171Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T01:48:47.7320191Z 
2022-11-23T01:48:47.7320458Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7320571Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7320590Z 
2022-11-23T01:48:47.7320704Z OK (skipped=1)
2022-11-23T01:48:47.7320724Z 
2022-11-23T01:48:47.7320827Z Generating XML reports...
2022-11-23T01:48:47.7321275Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014704.xml
2022-11-23T01:48:47.7321731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7321914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7322295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7322485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7322505Z 
2022-11-23T01:48:47.7322615Z Running tests...
2022-11-23T01:48:47.7322881Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7323196Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7323468Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:48:47.7323509Z 
2022-11-23T01:48:47.7323750Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7323867Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7323886Z 
2022-11-23T01:48:47.7323996Z OK (skipped=1)
2022-11-23T01:48:47.7324015Z 
2022-11-23T01:48:47.7324139Z Generating XML reports...
2022-11-23T01:48:47.7324584Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014706.xml
2022-11-23T01:48:47.7324956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7325131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7325512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7325742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7325761Z 
2022-11-23T01:48:47.7325870Z Running tests...
2022-11-23T01:48:47.7326138Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7326448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7326698Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7326918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19704
2022-11-23T01:48:47.7327136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19705
2022-11-23T01:48:47.7327507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7327661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7328042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7328233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7328596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7328771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7329144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7329331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7329579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7329822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7330204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7330606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7330884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7331115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7331219Z ok (5.941s)
2022-11-23T01:48:47.7331239Z 
2022-11-23T01:48:47.7331508Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7331622Z Ran 1 test in 5.941s
2022-11-23T01:48:47.7331642Z 
2022-11-23T01:48:47.7331735Z OK
2022-11-23T01:48:47.7331754Z 
2022-11-23T01:48:47.7331879Z Generating XML reports...
2022-11-23T01:48:47.7332309Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014709.xml
2022-11-23T01:48:47.7332682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7332858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7333241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7333433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7333452Z 
2022-11-23T01:48:47.7333563Z Running tests...
2022-11-23T01:48:47.7333826Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7334140Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7334397Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7334613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19822
2022-11-23T01:48:47.7334885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19823
2022-11-23T01:48:47.7335254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7335432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7335809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7335999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7336361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7336536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7336888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7337081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7337329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7337578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7338192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7338599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7338829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7339057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7339396Z STAGE:2022-11-23 01:47:23 19822:19822 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7339705Z STAGE:2022-11-23 01:47:23 19823:19823 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7340040Z STAGE:2022-11-23 01:47:23 19823:19823 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7340438Z STAGE:2022-11-23 01:47:23 19822:19822 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7340797Z STAGE:2022-11-23 01:47:23 19823:19823 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7341141Z STAGE:2022-11-23 01:47:23 19822:19822 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7341247Z ok (6.456s)
2022-11-23T01:48:47.7341267Z 
2022-11-23T01:48:47.7341529Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7341643Z Ran 1 test in 6.456s
2022-11-23T01:48:47.7341662Z 
2022-11-23T01:48:47.7341737Z OK
2022-11-23T01:48:47.7341756Z 
2022-11-23T01:48:47.7341880Z Generating XML reports...
2022-11-23T01:48:47.7342337Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014717.xml
2022-11-23T01:48:47.7342710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7342890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7343269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7343460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7343480Z 
2022-11-23T01:48:47.7343590Z Running tests...
2022-11-23T01:48:47.7343856Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7344149Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7344422Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7344713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19944
2022-11-23T01:48:47.7344929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19945
2022-11-23T01:48:47.7345305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7345481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7345860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7346052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7346396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7346573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7346954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7347144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7347391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7347631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7348034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7348431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7348662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7348869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7349212Z STAGE:2022-11-23 01:47:32 19945:19945 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7349537Z STAGE:2022-11-23 01:47:32 19944:19944 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:47.7349918Z STAGE:2022-11-23 01:47:32 19945:19945 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7350255Z STAGE:2022-11-23 01:47:32 19944:19944 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:47.7350602Z STAGE:2022-11-23 01:47:32 19945:19945 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7350948Z STAGE:2022-11-23 01:47:32 19944:19944 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:47.7351054Z ok (6.447s)
2022-11-23T01:48:47.7351074Z 
2022-11-23T01:48:47.7351338Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7351433Z Ran 1 test in 6.448s
2022-11-23T01:48:47.7351457Z 
2022-11-23T01:48:47.7351553Z OK
2022-11-23T01:48:47.7351573Z 
2022-11-23T01:48:47.7351697Z Generating XML reports...
2022-11-23T01:48:47.7352150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014726.xml
2022-11-23T01:48:47.7352523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7352702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7353082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7353272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7353292Z 
2022-11-23T01:48:47.7353383Z Running tests...
2022-11-23T01:48:47.7353649Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7353959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7354298Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:48:47.7354319Z 
2022-11-23T01:48:47.7354586Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7354701Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7354720Z 
2022-11-23T01:48:47.7354830Z OK (skipped=1)
2022-11-23T01:48:47.7354848Z 
2022-11-23T01:48:47.7354974Z Generating XML reports...
2022-11-23T01:48:47.7355418Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014735.xml
2022-11-23T01:48:47.7355763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7355942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7356318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7356511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7356530Z 
2022-11-23T01:48:47.7356641Z Running tests...
2022-11-23T01:48:47.7356908Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7357222Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7357496Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:48:47.7357515Z 
2022-11-23T01:48:47.7357773Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7357869Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7357889Z 
2022-11-23T01:48:47.7357997Z OK (skipped=1)
2022-11-23T01:48:47.7358016Z 
2022-11-23T01:48:47.7358140Z Generating XML reports...
2022-11-23T01:48:47.7358581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014737.xml
2022-11-23T01:48:47.7358952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7359178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7359564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7359753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7359773Z 
2022-11-23T01:48:47.7359883Z Running tests...
2022-11-23T01:48:47.7360126Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7360437Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7360738Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:48:47.7360762Z 
2022-11-23T01:48:47.7361019Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7361134Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7361153Z 
2022-11-23T01:48:47.7361262Z OK (skipped=1)
2022-11-23T01:48:47.7361284Z 
2022-11-23T01:48:47.7361411Z Generating XML reports...
2022-11-23T01:48:47.7361856Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014740.xml
2022-11-23T01:48:47.7362224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7362381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7362762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7362953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7363020Z 
2022-11-23T01:48:47.7363132Z Running tests...
2022-11-23T01:48:47.7363398Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7363710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7364008Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:48:47.7364028Z 
2022-11-23T01:48:47.7364288Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7364382Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7364421Z 
2022-11-23T01:48:47.7364511Z OK (skipped=1)
2022-11-23T01:48:47.7364530Z 
2022-11-23T01:48:47.7364655Z Generating XML reports...
2022-11-23T01:48:47.7365105Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014742.xml
2022-11-23T01:48:47.7365476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7365655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7366036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7366226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7366245Z 
2022-11-23T01:48:47.7366357Z Running tests...
2022-11-23T01:48:47.7366601Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7366916Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7367199Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-11-23T01:48:47.7367219Z 
2022-11-23T01:48:47.7367478Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7367599Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7367619Z 
2022-11-23T01:48:47.7367727Z OK (skipped=1)
2022-11-23T01:48:47.7367747Z 
2022-11-23T01:48:47.7367872Z Generating XML reports...
2022-11-23T01:48:47.7368376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014745.xml
2022-11-23T01:48:47.7368756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7368911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7369287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7369477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7369497Z 
2022-11-23T01:48:47.7369608Z Running tests...
2022-11-23T01:48:47.7369872Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7370190Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7370477Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-11-23T01:48:47.7370497Z 
2022-11-23T01:48:47.7370762Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7370876Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7370895Z 
2022-11-23T01:48:47.7370985Z OK (skipped=1)
2022-11-23T01:48:47.7371004Z 
2022-11-23T01:48:47.7371128Z Generating XML reports...
2022-11-23T01:48:47.7371574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014747.xml
2022-11-23T01:48:47.7371941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7372116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7372547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7372739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7372759Z 
2022-11-23T01:48:47.7372874Z Running tests...
2022-11-23T01:48:47.7373120Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7373433Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7373699Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7373919Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20264
2022-11-23T01:48:47.7374138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20265
2022-11-23T01:48:47.7374507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7374688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7375068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7375262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7375606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7375778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7376150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7376339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7376585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7376828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7377232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7377682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7378067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7378306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7378409Z ok (7.366s)
2022-11-23T01:48:47.7378429Z 
2022-11-23T01:48:47.7378697Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7378812Z Ran 1 test in 7.366s
2022-11-23T01:48:47.7378832Z 
2022-11-23T01:48:47.7378927Z OK
2022-11-23T01:48:47.7378946Z 
2022-11-23T01:48:47.7379073Z Generating XML reports...
2022-11-23T01:48:47.7379522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014749.xml
2022-11-23T01:48:47.7379893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7380053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7380429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7380619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7380638Z 
2022-11-23T01:48:47.7380748Z Running tests...
2022-11-23T01:48:47.7381010Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7381325Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7381600Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T01:48:47.7381698Z 
2022-11-23T01:48:47.7381966Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7382079Z Ran 1 test in 0.002s
2022-11-23T01:48:47.7382099Z 
2022-11-23T01:48:47.7382189Z OK (skipped=1)
2022-11-23T01:48:47.7382212Z 
2022-11-23T01:48:47.7382335Z Generating XML reports...
2022-11-23T01:48:47.7382830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014759.xml
2022-11-23T01:48:47.7383197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7383371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7383750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7383939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7383963Z 
2022-11-23T01:48:47.7384075Z Running tests...
2022-11-23T01:48:47.7384320Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7384636Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7384889Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7385109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20416
2022-11-23T01:48:47.7385327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20417
2022-11-23T01:48:47.7385696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7385874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7386253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7386446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7386789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7387033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7387421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7387609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7387853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7388094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7388497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7388899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7389128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7389340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7389598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeksgcxlu
2022-11-23T01:48:47.7389871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeksgcxlu/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7390128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7s9z1yw
2022-11-23T01:48:47.7390392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7s9z1yw/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7390498Z ok (6.869s)
2022-11-23T01:48:47.7390518Z 
2022-11-23T01:48:47.7390784Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7390951Z Ran 1 test in 6.869s
2022-11-23T01:48:47.7390970Z 
2022-11-23T01:48:47.7391044Z OK
2022-11-23T01:48:47.7391064Z 
2022-11-23T01:48:47.7391187Z Generating XML reports...
2022-11-23T01:48:47.7391645Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014802.xml
2022-11-23T01:48:47.7392017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7392194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7392570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7392760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7392779Z 
2022-11-23T01:48:47.7392890Z Running tests...
2022-11-23T01:48:47.7393150Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7393450Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7393742Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7393963Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20531
2022-11-23T01:48:47.7394180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20532
2022-11-23T01:48:47.7394547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7394724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7395099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7395289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7395636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7395810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7396227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7396421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7396665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7396910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7397312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7397706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7397943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7398150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7398412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp215lasb3
2022-11-23T01:48:47.7398683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp215lasb3/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7398934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3rqc6wg
2022-11-23T01:48:47.7399201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3rqc6wg/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7400012Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:47.7400835Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:47.7400941Z ok (7.444s)
2022-11-23T01:48:47.7400961Z 
2022-11-23T01:48:47.7401228Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7401348Z Ran 1 test in 7.444s
2022-11-23T01:48:47.7401368Z 
2022-11-23T01:48:47.7401462Z OK
2022-11-23T01:48:47.7401481Z 
2022-11-23T01:48:47.7401606Z Generating XML reports...
2022-11-23T01:48:47.7402041Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014811.xml
2022-11-23T01:48:47.7402420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7402596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7402975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7403167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7403187Z 
2022-11-23T01:48:47.7403299Z Running tests...
2022-11-23T01:48:47.7403561Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7403881Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7404161Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7404406Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20650
2022-11-23T01:48:47.7404630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20651
2022-11-23T01:48:47.7405002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7405177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7405556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7405747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7406112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7406291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7406646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7406836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7407080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7407325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7407723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7408119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7408401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7408628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7408869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.7409087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.7409488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.7409878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.7410118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:47.7410353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:47.7410744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.7411143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.7411403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsf6hjvi
2022-11-23T01:48:47.7411676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsf6hjvi/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7411909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjnnus6v
2022-11-23T01:48:47.7412173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjnnus6v/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7412280Z ok (11.330s)
2022-11-23T01:48:47.7412300Z 
2022-11-23T01:48:47.7412569Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7412685Z Ran 1 test in 11.330s
2022-11-23T01:48:47.7412708Z 
2022-11-23T01:48:47.7412803Z OK
2022-11-23T01:48:47.7412823Z 
2022-11-23T01:48:47.7412949Z Generating XML reports...
2022-11-23T01:48:47.7413401Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014821.xml
2022-11-23T01:48:47.7413826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7413991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7414372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7414565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7414585Z 
2022-11-23T01:48:47.7414698Z Running tests...
2022-11-23T01:48:47.7414962Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7415277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:48:47.7415568Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:47.7415790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20772
2022-11-23T01:48:47.7415988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20773
2022-11-23T01:48:47.7416356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7416531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7416915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7417195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7417558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:47.7417783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:47.7418334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:47.7418529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:47.7418755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:47.7418998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:47.7419403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7419799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:47.7420029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:47.7420262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:47.7420500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:47.7420744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:47.7421139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.7421511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:47.7421748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:47.7421982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:47.7422374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.7422766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:47.7423096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jlgaspa
2022-11-23T01:48:47.7423375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jlgaspa/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7423629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotoxnc51
2022-11-23T01:48:47.7423895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotoxnc51/_remote_module_non_scriptable.py
2022-11-23T01:48:47.7423980Z ok (11.262s)
2022-11-23T01:48:47.7424000Z 
2022-11-23T01:48:47.7424274Z ----------------------------------------------------------------------
2022-11-23T01:48:47.7424390Z Ran 1 test in 11.262s
2022-11-23T01:48:47.7424414Z 
2022-11-23T01:48:47.7424508Z OK
2022-11-23T01:48:47.7424528Z 
2022-11-23T01:48:47.7424652Z Generating XML reports...
2022-11-23T01:48:47.7425101Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014835.xml
2022-11-23T01:48:47.7425123Z 
2022-11-23T01:48:47.7425559Z ##[endgroup]
2022-11-23T01:48:47.7426012Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_u1jzs9hx)
2022-11-23T01:48:47.7426053Z 
2022-11-23T01:48:47.7426245Z Running distributed tests for the nccl backend with file init_method in shard 2 of 3
2022-11-23T01:48:47.7426756Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:48:47.535405]
2022-11-23T02:14:29.5030130Z 
2022-11-23T02:14:29.5030588Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T02:14:29.5033570Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_c7lcjva8)
2022-11-23T02:14:29.5034353Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:14:29.5070421Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T02:14:29.5122655Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5123614Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5124448Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5155586Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5156057Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5156552Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5157019Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5157517Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5158204Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5158748Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5159301Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5159832Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5160368Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5160885Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5161338Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5161839Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5162304Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5162717Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5163146Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5163605Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5164083Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5164542Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5164945Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5165348Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5165761Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5166186Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5166608Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5167037Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5167457Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5167927Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5168334Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5168718Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5169110Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5169523Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5169942Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5170338Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5170748Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5171177Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5171573Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5171979Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5172398Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5172821Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5173258Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5173710Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5174140Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5174551Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5174990Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5175500Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5175892Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5176342Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5176784Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5177198Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5177592Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5178485Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5178924Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5179312Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5179728Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5180158Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5180553Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5180929Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5181339Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5181736Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5182097Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5182479Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5182862Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5183249Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5183656Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5184048Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5184444Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5184810Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5185204Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5185671Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5186063Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5186475Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5186871Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5187225Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5187604Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5188001Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5188401Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5188781Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5189195Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5189581Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5189973Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5190400Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5190838Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5191283Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5191717Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5192165Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5192660Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5193158Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5193599Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5194035Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5194484Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5194906Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5195362Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5195828Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5196257Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5196712Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5197140Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5197535Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5197904Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5198268Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5198644Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5199005Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5199397Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5199796Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5200159Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5200566Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5200971Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5201373Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5201762Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5202169Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5202604Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5203043Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5203474Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5203891Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5204304Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5204717Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5205149Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5205559Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5205925Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5206308Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5206733Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5207108Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5207498Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5207891Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5208350Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5208841Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5209304Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5209718Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5210114Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5210597Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5211042Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5211514Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5211929Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5212349Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5212785Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5213173Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5213543Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5213928Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5214341Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5214738Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5215170Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5215606Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5216007Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5216421Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5216877Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5217374Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5218300Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5218929Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5219547Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5220222Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5220817Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5221417Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5222015Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5222612Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5223137Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5223632Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5224076Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5224445Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5224841Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5225248Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5225644Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5226045Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5226481Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5226938Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5227452Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5227865Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5228263Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5228678Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5229092Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5229516Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5229929Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5230320Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5230746Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5231172Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5231579Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5231984Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5232394Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5232809Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5233188Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5233604Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5234059Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5234475Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5234870Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5235250Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5235653Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5236048Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5236460Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5236887Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5237238Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5237610Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5237990Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5238347Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5238716Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5239098Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5239481Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5239826Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5240186Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5240563Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5240938Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5241330Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5241695Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5242022Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5242393Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5242795Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5243186Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5243633Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5244077Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5244540Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5244937Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5245367Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5245797Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5246195Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5246609Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5247016Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5247421Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5247809Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5248200Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5248594Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5249042Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5249529Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5249997Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5250442Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5250883Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5251330Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5251755Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5252151Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5252577Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5253013Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5253438Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5253962Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5254480Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5254958Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5255356Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5255753Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5256154Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5256536Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5256927Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5257303Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5257691Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5258400Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5258770Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5259130Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5259486Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5259865Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5260258Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5260646Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5261020Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5261390Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5261850Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5262217Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5262581Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5262949Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5263303Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5263679Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5264059Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5264432Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5264811Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5265192Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5265566Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5265922Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5266347Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5266791Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5267200Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5267598Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5268000Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5268406Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5268821Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5269214Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5269625Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5270036Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5270455Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5270861Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5271245Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5271703Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5272094Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5272493Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5272943Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5273380Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5274115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5274547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5275127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5275595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5275826Z 
2022-11-23T02:14:29.5275937Z Running tests...
2022-11-23T02:14:29.5276331Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5276859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5277447Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5277988Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20927
2022-11-23T02:14:29.5278439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20928
2022-11-23T02:14:29.5279042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5279550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5280103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5280574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5281147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5281567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5282135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5282596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5283043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5283524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5284172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5284858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5285380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5285828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5286344Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:14:29.5287171Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:14:29.5287837Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:14:29.5288626Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:14:29.5289324Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:14:29.5290142Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:14:29.5290793Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:14:29.5291581Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:14:29.5292235Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:14:29.5293091Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:14:29.5293748Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:14:29.5294539Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:14:29.5295192Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:14:29.5296000Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:14:29.5296713Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:14:29.5297503Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:14:29.5298385Z ok (6.963s)
2022-11-23T02:14:29.5298540Z 
2022-11-23T02:14:29.5298813Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5299142Z Ran 1 test in 6.963s
2022-11-23T02:14:29.5299281Z 
2022-11-23T02:14:29.5299383Z OK
2022-11-23T02:14:29.5299513Z 
2022-11-23T02:14:29.5299639Z Generating XML reports...
2022-11-23T02:14:29.5300248Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014851.xml
2022-11-23T02:14:29.5300943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5301394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5301968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5302435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5302646Z 
2022-11-23T02:14:29.5302758Z Running tests...
2022-11-23T02:14:29.5303159Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5303686Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5304194Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-11-23T02:14:29.5304498Z 
2022-11-23T02:14:29.5304763Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5305086Z Ran 1 test in 0.004s
2022-11-23T02:14:29.5305246Z 
2022-11-23T02:14:29.5305356Z OK (skipped=1)
2022-11-23T02:14:29.5305495Z 
2022-11-23T02:14:29.5305620Z Generating XML reports...
2022-11-23T02:14:29.5306221Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014901.xml
2022-11-23T02:14:29.5306998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5307437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5308008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5308475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5308703Z 
2022-11-23T02:14:29.5308813Z Running tests...
2022-11-23T02:14:29.5309192Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5309719Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5310232Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5310720Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21075
2022-11-23T02:14:29.5311152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21076
2022-11-23T02:14:29.5311752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5312197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5312746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5313208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5313778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5314219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5314836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5315293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5315744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5316224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5316876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5317561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5318078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5318523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5318863Z ok (4.239s)
2022-11-23T02:14:29.5319012Z 
2022-11-23T02:14:29.5319278Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5319588Z Ran 1 test in 4.240s
2022-11-23T02:14:29.5319751Z 
2022-11-23T02:14:29.5319849Z OK
2022-11-23T02:14:29.5319984Z 
2022-11-23T02:14:29.5320108Z Generating XML reports...
2022-11-23T02:14:29.5320713Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014903.xml
2022-11-23T02:14:29.5321404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5321845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5322417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5322890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5323099Z 
2022-11-23T02:14:29.5323210Z Running tests...
2022-11-23T02:14:29.5323612Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5324187Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5324699Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5325737Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-11-23T02:14:29.5326258Z 
2022-11-23T02:14:29.5326523Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5326856Z Ran 1 test in 1.640s
2022-11-23T02:14:29.5327021Z 
2022-11-23T02:14:29.5327131Z OK (skipped=1)
2022-11-23T02:14:29.5327266Z 
2022-11-23T02:14:29.5327391Z Generating XML reports...
2022-11-23T02:14:29.5327994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014910.xml
2022-11-23T02:14:29.5328710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5329139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5329710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5330175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5330403Z 
2022-11-23T02:14:29.5330514Z Running tests...
2022-11-23T02:14:29.5330897Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5331429Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5332044Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T02:14:29.5332368Z 
2022-11-23T02:14:29.5332636Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5332942Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5333104Z 
2022-11-23T02:14:29.5333214Z OK (skipped=1)
2022-11-23T02:14:29.5333366Z 
2022-11-23T02:14:29.5333490Z Generating XML reports...
2022-11-23T02:14:29.5334071Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014914.xml
2022-11-23T02:14:29.5334781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5335225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5335794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5336239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5336465Z 
2022-11-23T02:14:29.5336580Z Running tests...
2022-11-23T02:14:29.5336983Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5337489Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5338389Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T02:14:29.5338734Z 
2022-11-23T02:14:29.5338999Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5339319Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5339460Z 
2022-11-23T02:14:29.5339571Z OK (skipped=1)
2022-11-23T02:14:29.5339724Z 
2022-11-23T02:14:29.5339849Z Generating XML reports...
2022-11-23T02:14:29.5340452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014917.xml
2022-11-23T02:14:29.5341138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5341659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5342238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5342701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5342927Z 
2022-11-23T02:14:29.5343019Z Running tests...
2022-11-23T02:14:29.5343421Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5343947Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5344473Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5344999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21280
2022-11-23T02:14:29.5345444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21281
2022-11-23T02:14:29.5346051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5346481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5347046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5347513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5348084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5348503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5349063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5349593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5350027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5350526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5351178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5351863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5352365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5352834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5353333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94q_7nc4
2022-11-23T02:14:29.5353869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94q_7nc4/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5354374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf6rxfb0
2022-11-23T02:14:29.5354906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf6rxfb0/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5355806Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5356354Z   warnings.warn(
2022-11-23T02:14:29.5357090Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5357628Z   warnings.warn(
2022-11-23T02:14:29.5358002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5358513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5359003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5359481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5359954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5360404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5360876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5361346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5361800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5362272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5362740Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5363210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5363534Z ok (8.139s)
2022-11-23T02:14:29.5363682Z 
2022-11-23T02:14:29.5363953Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5364282Z Ran 1 test in 8.139s
2022-11-23T02:14:29.5364443Z 
2022-11-23T02:14:29.5364517Z OK
2022-11-23T02:14:29.5364651Z 
2022-11-23T02:14:29.5364776Z Generating XML reports...
2022-11-23T02:14:29.5365380Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014919.xml
2022-11-23T02:14:29.5366151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5366580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5367156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5367621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5367848Z 
2022-11-23T02:14:29.5367940Z Running tests...
2022-11-23T02:14:29.5368344Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5368873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5369430Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5369941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21399
2022-11-23T02:14:29.5370394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21400
2022-11-23T02:14:29.5370996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5371451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5372004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5372471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5373044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5373466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5374034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5374499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5374950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5375475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5376135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5376815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5377315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5377780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5378693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj21sd3jz
2022-11-23T02:14:29.5379239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj21sd3jz/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5379750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppc6t9a0b
2022-11-23T02:14:29.5380283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppc6t9a0b/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5381184Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5381731Z   warnings.warn(
2022-11-23T02:14:29.5382468Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5383003Z   warnings.warn(
2022-11-23T02:14:29.5383379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5383946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5384407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5384885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5385233Z ok (7.065s)
2022-11-23T02:14:29.5385380Z 
2022-11-23T02:14:29.5385634Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5385960Z Ran 1 test in 7.065s
2022-11-23T02:14:29.5386121Z 
2022-11-23T02:14:29.5386215Z OK
2022-11-23T02:14:29.5386349Z 
2022-11-23T02:14:29.5386473Z Generating XML reports...
2022-11-23T02:14:29.5387055Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014930.xml
2022-11-23T02:14:29.5387760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5388198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5388750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5389215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5389442Z 
2022-11-23T02:14:29.5389553Z Running tests...
2022-11-23T02:14:29.5389953Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5390458Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5391018Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5391557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21518
2022-11-23T02:14:29.5391984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21519
2022-11-23T02:14:29.5392616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5393061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5393696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5394145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5394721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5395159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5395706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5396161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5396614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5397107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5397737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5398414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5398930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5399398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5399874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_j463ks9
2022-11-23T02:14:29.5400401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_j463ks9/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5400984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkf2i7vfc
2022-11-23T02:14:29.5401497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkf2i7vfc/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5402399Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5402937Z   warnings.warn(
2022-11-23T02:14:29.5403690Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5404223Z   warnings.warn(
2022-11-23T02:14:29.5404577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5405061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5405535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5405992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5406462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5406927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5407397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5407849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5408189Z ok (7.169s)
2022-11-23T02:14:29.5408338Z 
2022-11-23T02:14:29.5408607Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5408920Z Ran 1 test in 7.170s
2022-11-23T02:14:29.5409080Z 
2022-11-23T02:14:29.5409175Z OK
2022-11-23T02:14:29.5409308Z 
2022-11-23T02:14:29.5409433Z Generating XML reports...
2022-11-23T02:14:29.5410064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014939.xml
2022-11-23T02:14:29.5410779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5411226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5411793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5412240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5412468Z 
2022-11-23T02:14:29.5412578Z Running tests...
2022-11-23T02:14:29.5412975Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5413506Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5414074Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5414639Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21637
2022-11-23T02:14:29.5415085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21638
2022-11-23T02:14:29.5415668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5416113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5416681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5417144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5417766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5418472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5419048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5419508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5419938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5420428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5421078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5421743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5422268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5422739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5423237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv0nunud
2022-11-23T02:14:29.5423754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv0nunud/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5424283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1t64lh6
2022-11-23T02:14:29.5424808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1t64lh6/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5425745Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5426277Z   warnings.warn(
2022-11-23T02:14:29.5427038Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5427645Z   warnings.warn(
2022-11-23T02:14:29.5428008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5428485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5428832Z ok (7.120s)
2022-11-23T02:14:29.5428981Z 
2022-11-23T02:14:29.5429246Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5429552Z Ran 1 test in 7.120s
2022-11-23T02:14:29.5429713Z 
2022-11-23T02:14:29.5429808Z OK
2022-11-23T02:14:29.5429941Z 
2022-11-23T02:14:29.5430065Z Generating XML reports...
2022-11-23T02:14:29.5430651Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014949.xml
2022-11-23T02:14:29.5431365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5431807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5432380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5432826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5433054Z 
2022-11-23T02:14:29.5433164Z Running tests...
2022-11-23T02:14:29.5433567Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5434072Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5434651Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5435277Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21756
2022-11-23T02:14:29.5435723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21757
2022-11-23T02:14:29.5436309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5436751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5437323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5437788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5438340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5438777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5439341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5439787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5440230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5440724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5441374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5442038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5442553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5443021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5443517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc1e5it4
2022-11-23T02:14:29.5444033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc1e5it4/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5444604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbx8ripew
2022-11-23T02:14:29.5445138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbx8ripew/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5446017Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5446560Z   warnings.warn(
2022-11-23T02:14:29.5447317Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5447856Z   warnings.warn(
2022-11-23T02:14:29.5448207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5448684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5449162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5449640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5449966Z ok (7.662s)
2022-11-23T02:14:29.5450110Z 
2022-11-23T02:14:29.5450380Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5450707Z Ran 1 test in 7.662s
2022-11-23T02:14:29.5450868Z 
2022-11-23T02:14:29.5450943Z OK
2022-11-23T02:14:29.5451077Z 
2022-11-23T02:14:29.5451201Z Generating XML reports...
2022-11-23T02:14:29.5451804Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014959.xml
2022-11-23T02:14:29.5452570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5452998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5453571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5454034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5454253Z 
2022-11-23T02:14:29.5454344Z Running tests...
2022-11-23T02:14:29.5454733Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5455239Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5455791Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5456314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21875
2022-11-23T02:14:29.5456742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21876
2022-11-23T02:14:29.5457319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5457749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5458543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5458991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5459560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5459979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5460541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5461008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5461454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5462006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5462661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5463348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5463870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5464319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5464811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoeucma5c
2022-11-23T02:14:29.5465354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoeucma5c/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5465864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9hje_nh
2022-11-23T02:14:29.5466395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9hje_nh/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5467289Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5467835Z   warnings.warn(
2022-11-23T02:14:29.5468566Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5469177Z   warnings.warn(
2022-11-23T02:14:29.5469552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5470033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5470490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5470968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5471314Z ok (7.329s)
2022-11-23T02:14:29.5471462Z 
2022-11-23T02:14:29.5471716Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5472044Z Ran 1 test in 7.329s
2022-11-23T02:14:29.5472204Z 
2022-11-23T02:14:29.5472297Z OK
2022-11-23T02:14:29.5472432Z 
2022-11-23T02:14:29.5472557Z Generating XML reports...
2022-11-23T02:14:29.5473145Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015009.xml
2022-11-23T02:14:29.5473860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5474309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5474865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5475332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5475561Z 
2022-11-23T02:14:29.5475670Z Running tests...
2022-11-23T02:14:29.5476071Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5476577Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5477153Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5477711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21994
2022-11-23T02:14:29.5478147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21995
2022-11-23T02:14:29.5478748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5479246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5479823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5480270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5480840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5481281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5481845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5482289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5482737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5483233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5483863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5484539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5485057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5485524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5485998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dj9bpet
2022-11-23T02:14:29.5486588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dj9bpet/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5487116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4p7g_4yf
2022-11-23T02:14:29.5487649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4p7g_4yf/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5488529Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5489071Z   warnings.warn(
2022-11-23T02:14:29.5489823Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.5490362Z   warnings.warn(
2022-11-23T02:14:29.5490718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5491199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5491674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5492130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5492471Z ok (7.009s)
2022-11-23T02:14:29.5492657Z 
2022-11-23T02:14:29.5492925Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5493252Z Ran 1 test in 7.009s
2022-11-23T02:14:29.5493393Z 
2022-11-23T02:14:29.5493487Z OK
2022-11-23T02:14:29.5493622Z 
2022-11-23T02:14:29.5493748Z Generating XML reports...
2022-11-23T02:14:29.5494350Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015019.xml
2022-11-23T02:14:29.5495039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5495488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5496107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5496581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5496789Z 
2022-11-23T02:14:29.5496901Z Running tests...
2022-11-23T02:14:29.5497305Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5497832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5498589Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5499648Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T02:14:29.5500165Z 
2022-11-23T02:14:29.5500432Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5500761Z Ran 1 test in 1.642s
2022-11-23T02:14:29.5500922Z 
2022-11-23T02:14:29.5501012Z OK (skipped=1)
2022-11-23T02:14:29.5501166Z 
2022-11-23T02:14:29.5501292Z Generating XML reports...
2022-11-23T02:14:29.5501894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015028.xml
2022-11-23T02:14:29.5502605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5503034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5503601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5504148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5504377Z 
2022-11-23T02:14:29.5504489Z Running tests...
2022-11-23T02:14:29.5504877Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5505409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5505954Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5506459Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22147
2022-11-23T02:14:29.5506909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22148
2022-11-23T02:14:29.5507513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5507962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5508518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5508985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5509562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5509981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5510547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5511005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5511453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5511932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5512586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5513331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5513854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5514299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5514636Z ok (4.353s)
2022-11-23T02:14:29.5514783Z 
2022-11-23T02:14:29.5515047Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5515354Z Ran 1 test in 4.353s
2022-11-23T02:14:29.5515514Z 
2022-11-23T02:14:29.5515608Z OK
2022-11-23T02:14:29.5515740Z 
2022-11-23T02:14:29.5515866Z Generating XML reports...
2022-11-23T02:14:29.5516450Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015032.xml
2022-11-23T02:14:29.5517164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5517611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5518183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5518630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5518856Z 
2022-11-23T02:14:29.5518966Z Running tests...
2022-11-23T02:14:29.5519365Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5519898Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5520440Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5521545Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.650s)
2022-11-23T02:14:29.5522056Z 
2022-11-23T02:14:29.5522316Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5522641Z Ran 1 test in 1.650s
2022-11-23T02:14:29.5522781Z 
2022-11-23T02:14:29.5522890Z OK (skipped=1)
2022-11-23T02:14:29.5523043Z 
2022-11-23T02:14:29.5523167Z Generating XML reports...
2022-11-23T02:14:29.5523765Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015039.xml
2022-11-23T02:14:29.5524450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5524898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5525471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5525932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5526157Z 
2022-11-23T02:14:29.5526250Z Running tests...
2022-11-23T02:14:29.5526651Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5527174Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5527679Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5528182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22286
2022-11-23T02:14:29.5528626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22287
2022-11-23T02:14:29.5529229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5529657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5530220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5530730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5531309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5531726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5532285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5532743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5533167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5533667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5534315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5535000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5535500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5535966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5536309Z ok (6.766s)
2022-11-23T02:14:29.5536456Z 
2022-11-23T02:14:29.5536724Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5537032Z Ran 1 test in 6.767s
2022-11-23T02:14:29.5537192Z 
2022-11-23T02:14:29.5537287Z OK
2022-11-23T02:14:29.5537420Z 
2022-11-23T02:14:29.5537547Z Generating XML reports...
2022-11-23T02:14:29.5538421Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015043.xml
2022-11-23T02:14:29.5539139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5539584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5540153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5540601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5540828Z 
2022-11-23T02:14:29.5540939Z Running tests...
2022-11-23T02:14:29.5541341Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5541846Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5542344Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T02:14:29.5542635Z 
2022-11-23T02:14:29.5542898Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5543222Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5543362Z 
2022-11-23T02:14:29.5543475Z OK (skipped=1)
2022-11-23T02:14:29.5543629Z 
2022-11-23T02:14:29.5543754Z Generating XML reports...
2022-11-23T02:14:29.5544353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015053.xml
2022-11-23T02:14:29.5545045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5545493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5546060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5546522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5546733Z 
2022-11-23T02:14:29.5546843Z Running tests...
2022-11-23T02:14:29.5547245Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5547845Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5548286Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5548759Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5549236Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22434
2022-11-23T02:14:29.5549683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22435
2022-11-23T02:14:29.5550268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5550713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5551284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5551730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5552304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5552742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5553307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5553744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5554191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5554686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5555337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5556076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5556595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5557060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5557536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppuk9lzxb
2022-11-23T02:14:29.5558076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppuk9lzxb/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5558604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps675sfny
2022-11-23T02:14:29.5559129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps675sfny/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5559491Z ok (7.230s)
2022-11-23T02:14:29.5559640Z 
2022-11-23T02:14:29.5559910Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5560232Z Ran 1 test in 7.230s
2022-11-23T02:14:29.5560394Z 
2022-11-23T02:14:29.5560470Z OK
2022-11-23T02:14:29.5560604Z 
2022-11-23T02:14:29.5560730Z Generating XML reports...
2022-11-23T02:14:29.5561331Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015055.xml
2022-11-23T02:14:29.5562042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5562468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5563037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5563499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5563731Z 
2022-11-23T02:14:29.5563843Z Running tests...
2022-11-23T02:14:29.5564227Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5564751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5565273Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5565758Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5566237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22553
2022-11-23T02:14:29.5566682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22554
2022-11-23T02:14:29.5567282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5567710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5568277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5568750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5569310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5569753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5570318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5570780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5571209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5571705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5572353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5573078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5573603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5574071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5574569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc26gomwt
2022-11-23T02:14:29.5575086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc26gomwt/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5575612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgnymbh6
2022-11-23T02:14:29.5576146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgnymbh6/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5576657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5577122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5577474Z ok (7.468s)
2022-11-23T02:14:29.5577623Z 
2022-11-23T02:14:29.5578104Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5578423Z Ran 1 test in 7.468s
2022-11-23T02:14:29.5578585Z 
2022-11-23T02:14:29.5578680Z OK
2022-11-23T02:14:29.5578816Z 
2022-11-23T02:14:29.5578942Z Generating XML reports...
2022-11-23T02:14:29.5579551Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015105.xml
2022-11-23T02:14:29.5580239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5580686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5581256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5581708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5581936Z 
2022-11-23T02:14:29.5582047Z Running tests...
2022-11-23T02:14:29.5582521Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5583055Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5583529Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5584057Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5584556Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22672
2022-11-23T02:14:29.5584983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22673
2022-11-23T02:14:29.5585582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5586036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5586605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5587049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5587622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5588061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5588630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5589071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5589523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5590086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5590718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5591404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5591924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5592391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5592906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplt_m4rti
2022-11-23T02:14:29.5593440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplt_m4rti/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5593961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_0bou9q
2022-11-23T02:14:29.5594491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_0bou9q/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5594977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5595460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.5595813Z ok (7.415s)
2022-11-23T02:14:29.5595962Z 
2022-11-23T02:14:29.5596215Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5596545Z Ran 1 test in 7.415s
2022-11-23T02:14:29.5596707Z 
2022-11-23T02:14:29.5596803Z OK
2022-11-23T02:14:29.5596937Z 
2022-11-23T02:14:29.5597042Z Generating XML reports...
2022-11-23T02:14:29.5597644Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015115.xml
2022-11-23T02:14:29.5598351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5598803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5599352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5599867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5600103Z 
2022-11-23T02:14:29.5600214Z Running tests...
2022-11-23T02:14:29.5600619Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5601125Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5601593Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:14:29.5602085Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5602541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22791
2022-11-23T02:14:29.5602991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22792
2022-11-23T02:14:29.5603589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5604017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5604579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5605040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5605604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5606024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5606583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5607090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5607516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5608013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5608662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5609339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5609835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5610297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5610790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe14wqxfj
2022-11-23T02:14:29.5611324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe14wqxfj/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5611828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rt48d5e
2022-11-23T02:14:29.5612350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rt48d5e/_remote_module_non_scriptable.py
2022-11-23T02:14:29.5612721Z ok (7.367s)
2022-11-23T02:14:29.5612867Z 
2022-11-23T02:14:29.5613118Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5613440Z Ran 1 test in 7.367s
2022-11-23T02:14:29.5613599Z 
2022-11-23T02:14:29.5613690Z OK
2022-11-23T02:14:29.5613822Z 
2022-11-23T02:14:29.5613944Z Generating XML reports...
2022-11-23T02:14:29.5614528Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015125.xml
2022-11-23T02:14:29.5615230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5615674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5616224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5616737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5616969Z 
2022-11-23T02:14:29.5617079Z Running tests...
2022-11-23T02:14:29.5617481Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5618203Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5618702Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5619010Z 
2022-11-23T02:14:29.5619272Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5619577Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5619735Z 
2022-11-23T02:14:29.5619848Z OK (skipped=1)
2022-11-23T02:14:29.5620001Z 
2022-11-23T02:14:29.5620125Z Generating XML reports...
2022-11-23T02:14:29.5620722Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015135.xml
2022-11-23T02:14:29.5621411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5621848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5622411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5622876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5623085Z 
2022-11-23T02:14:29.5623192Z Running tests...
2022-11-23T02:14:29.5623589Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5624111Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5624711Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T02:14:29.5625027Z 
2022-11-23T02:14:29.5625290Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5625610Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5625767Z 
2022-11-23T02:14:29.5625875Z OK (skipped=1)
2022-11-23T02:14:29.5626010Z 
2022-11-23T02:14:29.5626133Z Generating XML reports...
2022-11-23T02:14:29.5626727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015137.xml
2022-11-23T02:14:29.5627427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5627851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5628414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5628876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5629101Z 
2022-11-23T02:14:29.5629210Z Running tests...
2022-11-23T02:14:29.5629593Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5630112Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5630644Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T02:14:29.5630959Z 
2022-11-23T02:14:29.5631217Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5631517Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5631673Z 
2022-11-23T02:14:29.5631781Z OK (skipped=1)
2022-11-23T02:14:29.5631934Z 
2022-11-23T02:14:29.5632057Z Generating XML reports...
2022-11-23T02:14:29.5632632Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015140.xml
2022-11-23T02:14:29.5633338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5633836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5634414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5634857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5635081Z 
2022-11-23T02:14:29.5635190Z Running tests...
2022-11-23T02:14:29.5635586Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5636086Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5636612Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T02:14:29.5636925Z 
2022-11-23T02:14:29.5637186Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5637503Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5637643Z 
2022-11-23T02:14:29.5637750Z OK (skipped=1)
2022-11-23T02:14:29.5637900Z 
2022-11-23T02:14:29.5638025Z Generating XML reports...
2022-11-23T02:14:29.5638620Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015142.xml
2022-11-23T02:14:29.5639308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5639749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5640313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5640774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5641052Z 
2022-11-23T02:14:29.5641144Z Running tests...
2022-11-23T02:14:29.5641547Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5642067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5642583Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T02:14:29.5642892Z 
2022-11-23T02:14:29.5643152Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5643468Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5643625Z 
2022-11-23T02:14:29.5643732Z OK (skipped=1)
2022-11-23T02:14:29.5643751Z 
2022-11-23T02:14:29.5643872Z Generating XML reports...
2022-11-23T02:14:29.5644297Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015144.xml
2022-11-23T02:14:29.5644659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5644838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5645212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5645402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5645422Z 
2022-11-23T02:14:29.5645529Z Running tests...
2022-11-23T02:14:29.5645793Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5646102Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5646373Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-11-23T02:14:29.5646409Z 
2022-11-23T02:14:29.5646649Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5646758Z Ran 1 test in 0.003s
2022-11-23T02:14:29.5646781Z 
2022-11-23T02:14:29.5646889Z OK (skipped=1)
2022-11-23T02:14:29.5646908Z 
2022-11-23T02:14:29.5647033Z Generating XML reports...
2022-11-23T02:14:29.5647476Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015147.xml
2022-11-23T02:14:29.5647890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5648069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5648447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5648618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5648655Z 
2022-11-23T02:14:29.5648745Z Running tests...
2022-11-23T02:14:29.5649001Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5649311Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5649579Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5649598Z 
2022-11-23T02:14:29.5649858Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5649969Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5649988Z 
2022-11-23T02:14:29.5650096Z OK (skipped=1)
2022-11-23T02:14:29.5650114Z 
2022-11-23T02:14:29.5650236Z Generating XML reports...
2022-11-23T02:14:29.5650659Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015149.xml
2022-11-23T02:14:29.5651020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5651191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5651560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5651810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5651829Z 
2022-11-23T02:14:29.5651937Z Running tests...
2022-11-23T02:14:29.5652199Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5652506Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5652763Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5652963Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23141
2022-11-23T02:14:29.5653177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23142
2022-11-23T02:14:29.5653544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5653718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5654099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5654286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5654647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5654818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5655169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5655356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5655595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5655838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5656238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5656630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5656904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5657134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5657468Z STAGE:2022-11-23 01:51:57 23141:23141 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5657780Z STAGE:2022-11-23 01:51:57 23142:23142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5658331Z STAGE:2022-11-23 01:51:58 23141:23141 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5658662Z STAGE:2022-11-23 01:51:58 23142:23142 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5659013Z STAGE:2022-11-23 01:51:58 23142:23142 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5659360Z STAGE:2022-11-23 01:51:58 23141:23141 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5659684Z STAGE:2022-11-23 01:51:59 23142:23142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5660015Z STAGE:2022-11-23 01:51:59 23142:23142 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5660353Z STAGE:2022-11-23 01:51:59 23142:23142 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5660678Z STAGE:2022-11-23 01:51:59 23141:23141 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5660986Z STAGE:2022-11-23 01:51:59 23141:23141 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5661324Z STAGE:2022-11-23 01:51:59 23141:23141 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5661504Z ok (7.778s)
2022-11-23T02:14:29.5661524Z 
2022-11-23T02:14:29.5661789Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5661901Z Ran 1 test in 7.778s
2022-11-23T02:14:29.5661924Z 
2022-11-23T02:14:29.5662018Z OK
2022-11-23T02:14:29.5662036Z 
2022-11-23T02:14:29.5662159Z Generating XML reports...
2022-11-23T02:14:29.5662607Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015152.xml
2022-11-23T02:14:29.5662975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5663132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5663506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5663699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5663719Z 
2022-11-23T02:14:29.5663826Z Running tests...
2022-11-23T02:14:29.5664087Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5664399Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5664661Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5664878Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23260
2022-11-23T02:14:29.5665073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23261
2022-11-23T02:14:29.5665440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5665611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5665982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5666171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5666588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5666771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5667141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5667307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5667547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5667786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5668178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5668575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5668808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5669032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5669367Z STAGE:2022-11-23 01:52:07 23261:23261 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5669691Z STAGE:2022-11-23 01:52:08 23260:23260 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5670004Z STAGE:2022-11-23 01:52:09 23260:23260 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5670326Z STAGE:2022-11-23 01:52:09 23261:23261 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5670663Z STAGE:2022-11-23 01:52:09 23260:23260 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5671060Z STAGE:2022-11-23 01:52:09 23261:23261 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5671383Z STAGE:2022-11-23 01:52:09 23261:23261 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5671713Z STAGE:2022-11-23 01:52:09 23261:23261 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5672054Z STAGE:2022-11-23 01:52:09 23261:23261 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5672376Z STAGE:2022-11-23 01:52:09 23260:23260 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5672704Z STAGE:2022-11-23 01:52:09 23260:23260 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5673027Z STAGE:2022-11-23 01:52:09 23260:23260 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5673133Z ok (7.764s)
2022-11-23T02:14:29.5673153Z 
2022-11-23T02:14:29.5673416Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5673528Z Ran 1 test in 7.764s
2022-11-23T02:14:29.5673547Z 
2022-11-23T02:14:29.5673638Z OK
2022-11-23T02:14:29.5673657Z 
2022-11-23T02:14:29.5673783Z Generating XML reports...
2022-11-23T02:14:29.5674233Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015202.xml
2022-11-23T02:14:29.5674605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5674778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5675137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5675325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5675348Z 
2022-11-23T02:14:29.5675456Z Running tests...
2022-11-23T02:14:29.5675716Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5676027Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5676337Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5676358Z 
2022-11-23T02:14:29.5676623Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5676735Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5676754Z 
2022-11-23T02:14:29.5676861Z OK (skipped=1)
2022-11-23T02:14:29.5676881Z 
2022-11-23T02:14:29.5676986Z Generating XML reports...
2022-11-23T02:14:29.5677433Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015212.xml
2022-11-23T02:14:29.5677799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5677976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5678348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5678537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5678556Z 
2022-11-23T02:14:29.5678663Z Running tests...
2022-11-23T02:14:29.5678921Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5679212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5679471Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5679490Z 
2022-11-23T02:14:29.5679738Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5679848Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5679913Z 
2022-11-23T02:14:29.5680024Z OK (skipped=1)
2022-11-23T02:14:29.5680043Z 
2022-11-23T02:14:29.5680166Z Generating XML reports...
2022-11-23T02:14:29.5680613Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015215.xml
2022-11-23T02:14:29.5680978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5681153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5681509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5681694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5681714Z 
2022-11-23T02:14:29.5681819Z Running tests...
2022-11-23T02:14:29.5682080Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5682390Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5682666Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5682882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23445
2022-11-23T02:14:29.5683099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23446
2022-11-23T02:14:29.5683448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5683621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5683993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5684180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5684536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5684709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5685077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5685311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5685557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5685779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5686178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5686573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5686800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5687028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5687359Z STAGE:2022-11-23 01:52:22 23445:23445 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5687685Z STAGE:2022-11-23 01:52:23 23446:23446 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5688020Z STAGE:2022-11-23 01:52:24 23445:23445 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5688352Z STAGE:2022-11-23 01:52:24 23446:23446 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5688678Z STAGE:2022-11-23 01:52:24 23445:23445 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5689020Z STAGE:2022-11-23 01:52:24 23446:23446 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5689122Z ok (7.855s)
2022-11-23T02:14:29.5689142Z 
2022-11-23T02:14:29.5689452Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5689563Z Ran 1 test in 7.855s
2022-11-23T02:14:29.5689583Z 
2022-11-23T02:14:29.5689674Z OK
2022-11-23T02:14:29.5689693Z 
2022-11-23T02:14:29.5689816Z Generating XML reports...
2022-11-23T02:14:29.5690264Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015217.xml
2022-11-23T02:14:29.5690626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5690782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5691157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5691343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5691363Z 
2022-11-23T02:14:29.5691471Z Running tests...
2022-11-23T02:14:29.5691737Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5692050Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5692328Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5692580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23564
2022-11-23T02:14:29.5692779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23565
2022-11-23T02:14:29.5693146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5693318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5693690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5693878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5694240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5694412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5694824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5695013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5695238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5695477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5695873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5696262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5696495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5696719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5697056Z STAGE:2022-11-23 01:52:33 23564:23564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5697377Z STAGE:2022-11-23 01:52:33 23565:23565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5697693Z STAGE:2022-11-23 01:52:34 23564:23564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5698220Z STAGE:2022-11-23 01:52:34 23565:23565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5698578Z STAGE:2022-11-23 01:52:34 23564:23564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5698923Z STAGE:2022-11-23 01:52:34 23565:23565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5699103Z ok (7.828s)
2022-11-23T02:14:29.5699123Z 
2022-11-23T02:14:29.5699383Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5699493Z Ran 1 test in 7.828s
2022-11-23T02:14:29.5699513Z 
2022-11-23T02:14:29.5699608Z OK
2022-11-23T02:14:29.5699628Z 
2022-11-23T02:14:29.5699751Z Generating XML reports...
2022-11-23T02:14:29.5700180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015227.xml
2022-11-23T02:14:29.5700544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5700716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5701089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5701277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5701300Z 
2022-11-23T02:14:29.5701408Z Running tests...
2022-11-23T02:14:29.5701668Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5701983Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5702239Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5702967Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.609s)
2022-11-23T02:14:29.5703006Z 
2022-11-23T02:14:29.5703250Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5703361Z Ran 1 test in 1.609s
2022-11-23T02:14:29.5703385Z 
2022-11-23T02:14:29.5703491Z OK (skipped=1)
2022-11-23T02:14:29.5703510Z 
2022-11-23T02:14:29.5703633Z Generating XML reports...
2022-11-23T02:14:29.5704073Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015238.xml
2022-11-23T02:14:29.5704498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5704677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5705053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5705225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5705262Z 
2022-11-23T02:14:29.5705353Z Running tests...
2022-11-23T02:14:29.5705609Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5705919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5706191Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5706931Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2022-11-23T02:14:29.5706952Z 
2022-11-23T02:14:29.5707210Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5707321Z Ran 1 test in 1.628s
2022-11-23T02:14:29.5707340Z 
2022-11-23T02:14:29.5707445Z OK (skipped=1)
2022-11-23T02:14:29.5707463Z 
2022-11-23T02:14:29.5707586Z Generating XML reports...
2022-11-23T02:14:29.5708013Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015242.xml
2022-11-23T02:14:29.5708433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5708606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5708984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5709171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5709191Z 
2022-11-23T02:14:29.5709298Z Running tests...
2022-11-23T02:14:29.5709558Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5709868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5710125Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5710342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23751
2022-11-23T02:14:29.5710561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23752
2022-11-23T02:14:29.5710930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5711105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5711478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5711665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5712021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5712192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5712540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5712729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5712970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5713249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5713655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5714050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5714277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5714502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5714601Z ok (6.851s)
2022-11-23T02:14:29.5714621Z 
2022-11-23T02:14:29.5714865Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5714980Z Ran 1 test in 6.851s
2022-11-23T02:14:29.5714999Z 
2022-11-23T02:14:29.5715090Z OK
2022-11-23T02:14:29.5715109Z 
2022-11-23T02:14:29.5715234Z Generating XML reports...
2022-11-23T02:14:29.5715680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015246.xml
2022-11-23T02:14:29.5716049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5716223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5716599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5716770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5716806Z 
2022-11-23T02:14:29.5716896Z Running tests...
2022-11-23T02:14:29.5717152Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5717527Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5717798Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5718017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23866
2022-11-23T02:14:29.5718230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23867
2022-11-23T02:14:29.5718594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5718769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5719122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5719307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5719665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5719840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5720210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5720392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5720631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5720870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5721249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5721641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5721871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5722094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5722389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.5722634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.5723026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.5723411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.5723645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:14:29.5723877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:14:29.5724255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.5724641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.5724877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T02:14:29.5725108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T02:14:29.5725496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:14:29.5725929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:14:29.5726032Z ok (8.469s)
2022-11-23T02:14:29.5726052Z 
2022-11-23T02:14:29.5726315Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5726461Z Ran 1 test in 8.469s
2022-11-23T02:14:29.5726497Z 
2022-11-23T02:14:29.5726572Z OK
2022-11-23T02:14:29.5726591Z 
2022-11-23T02:14:29.5726714Z Generating XML reports...
2022-11-23T02:14:29.5727165Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015255.xml
2022-11-23T02:14:29.5727535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5727708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5728083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5728270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5728290Z 
2022-11-23T02:14:29.5728397Z Running tests...
2022-11-23T02:14:29.5728641Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5728959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5729212Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5729435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24017
2022-11-23T02:14:29.5729645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24018
2022-11-23T02:14:29.5730012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5730184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5730559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5730747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5731096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5731268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5731682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5731872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5732114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5732356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5732753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5733146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5733359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5733581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5733682Z ok (6.955s)
2022-11-23T02:14:29.5733701Z 
2022-11-23T02:14:29.5733965Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5734076Z Ran 1 test in 6.955s
2022-11-23T02:14:29.5734095Z 
2022-11-23T02:14:29.5734187Z OK
2022-11-23T02:14:29.5734205Z 
2022-11-23T02:14:29.5734330Z Generating XML reports...
2022-11-23T02:14:29.5734775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015306.xml
2022-11-23T02:14:29.5735139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5735293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5735723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5735911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5735930Z 
2022-11-23T02:14:29.5736038Z Running tests...
2022-11-23T02:14:29.5736302Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5736614Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5737027Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5737046Z 
2022-11-23T02:14:29.5737300Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5737410Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5737429Z 
2022-11-23T02:14:29.5737519Z OK (skipped=1)
2022-11-23T02:14:29.5737538Z 
2022-11-23T02:14:29.5737660Z Generating XML reports...
2022-11-23T02:14:29.5738328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015316.xml
2022-11-23T02:14:29.5738711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5738884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5739253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5739441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5739461Z 
2022-11-23T02:14:29.5739569Z Running tests...
2022-11-23T02:14:29.5739810Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5740122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5740532Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5740557Z 
2022-11-23T02:14:29.5740815Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5740920Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5740940Z 
2022-11-23T02:14:29.5741116Z OK (skipped=1)
2022-11-23T02:14:29.5741139Z 
2022-11-23T02:14:29.5741268Z Generating XML reports...
2022-11-23T02:14:29.5741714Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015318.xml
2022-11-23T02:14:29.5742077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5742232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5742607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5742795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5742815Z 
2022-11-23T02:14:29.5742924Z Running tests...
2022-11-23T02:14:29.5743186Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5743498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5743917Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5743938Z 
2022-11-23T02:14:29.5744197Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5744310Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5744329Z 
2022-11-23T02:14:29.5744420Z OK (skipped=1)
2022-11-23T02:14:29.5744439Z 
2022-11-23T02:14:29.5744561Z Generating XML reports...
2022-11-23T02:14:29.5745003Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015321.xml
2022-11-23T02:14:29.5745436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5745609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5745987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5746174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5746194Z 
2022-11-23T02:14:29.5746300Z Running tests...
2022-11-23T02:14:29.5746559Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5746851Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5747262Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5747282Z 
2022-11-23T02:14:29.5747541Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5747653Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5747673Z 
2022-11-23T02:14:29.5747779Z OK (skipped=1)
2022-11-23T02:14:29.5747797Z 
2022-11-23T02:14:29.5747920Z Generating XML reports...
2022-11-23T02:14:29.5748364Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015323.xml
2022-11-23T02:14:29.5748731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5748904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5749262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5749448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5749468Z 
2022-11-23T02:14:29.5749574Z Running tests...
2022-11-23T02:14:29.5749836Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5750144Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5750596Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5750617Z 
2022-11-23T02:14:29.5750880Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5750994Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5751014Z 
2022-11-23T02:14:29.5751103Z OK (skipped=1)
2022-11-23T02:14:29.5751139Z 
2022-11-23T02:14:29.5751244Z Generating XML reports...
2022-11-23T02:14:29.5751684Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015325.xml
2022-11-23T02:14:29.5752047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5752227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5752601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5752785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5752807Z 
2022-11-23T02:14:29.5752914Z Running tests...
2022-11-23T02:14:29.5753176Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5753469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5753875Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5753895Z 
2022-11-23T02:14:29.5754152Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5754263Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5754282Z 
2022-11-23T02:14:29.5754436Z OK (skipped=1)
2022-11-23T02:14:29.5754456Z 
2022-11-23T02:14:29.5754578Z Generating XML reports...
2022-11-23T02:14:29.5755021Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015328.xml
2022-11-23T02:14:29.5755387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5755559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5755914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5756101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5756120Z 
2022-11-23T02:14:29.5756227Z Running tests...
2022-11-23T02:14:29.5756485Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5756793Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5757208Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5757229Z 
2022-11-23T02:14:29.5757483Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5757598Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5757617Z 
2022-11-23T02:14:29.5757725Z OK (skipped=1)
2022-11-23T02:14:29.5757744Z 
2022-11-23T02:14:29.5757850Z Generating XML reports...
2022-11-23T02:14:29.5758293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015330.xml
2022-11-23T02:14:29.5758656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5758829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5759203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5759393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5759412Z 
2022-11-23T02:14:29.5759519Z Running tests...
2022-11-23T02:14:29.5759777Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5760116Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5760530Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5760551Z 
2022-11-23T02:14:29.5760806Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5760916Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5760935Z 
2022-11-23T02:14:29.5761041Z OK (skipped=1)
2022-11-23T02:14:29.5761060Z 
2022-11-23T02:14:29.5761182Z Generating XML reports...
2022-11-23T02:14:29.5761620Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015333.xml
2022-11-23T02:14:29.5761989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5762161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5762519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5762706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5762726Z 
2022-11-23T02:14:29.5762833Z Running tests...
2022-11-23T02:14:29.5763090Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5763398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5763791Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5763857Z 
2022-11-23T02:14:29.5764120Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5764231Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5764250Z 
2022-11-23T02:14:29.5764357Z OK (skipped=1)
2022-11-23T02:14:29.5764376Z 
2022-11-23T02:14:29.5764485Z Generating XML reports...
2022-11-23T02:14:29.5764923Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015335.xml
2022-11-23T02:14:29.5765285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5765457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5765828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5766014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5766034Z 
2022-11-23T02:14:29.5766146Z Running tests...
2022-11-23T02:14:29.5766401Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5766707Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5766993Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5767013Z 
2022-11-23T02:14:29.5767267Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5767376Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5767395Z 
2022-11-23T02:14:29.5767502Z OK (skipped=1)
2022-11-23T02:14:29.5767520Z 
2022-11-23T02:14:29.5767643Z Generating XML reports...
2022-11-23T02:14:29.5768084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015337.xml
2022-11-23T02:14:29.5768447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5768623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5768997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5769212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5769234Z 
2022-11-23T02:14:29.5769347Z Running tests...
2022-11-23T02:14:29.5769606Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5769913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5770302Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5770322Z 
2022-11-23T02:14:29.5770576Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5770689Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5770713Z 
2022-11-23T02:14:29.5770820Z OK (skipped=1)
2022-11-23T02:14:29.5770839Z 
2022-11-23T02:14:29.5770944Z Generating XML reports...
2022-11-23T02:14:29.5771383Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015340.xml
2022-11-23T02:14:29.5771749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5771919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5772293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5772478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5772498Z 
2022-11-23T02:14:29.5772606Z Running tests...
2022-11-23T02:14:29.5772862Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5773170Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5773607Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5773644Z 
2022-11-23T02:14:29.5773889Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5773999Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5774018Z 
2022-11-23T02:14:29.5774124Z OK (skipped=1)
2022-11-23T02:14:29.5774143Z 
2022-11-23T02:14:29.5774265Z Generating XML reports...
2022-11-23T02:14:29.5774705Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015342.xml
2022-11-23T02:14:29.5775068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5775243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5775616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5775791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5775810Z 
2022-11-23T02:14:29.5775917Z Running tests...
2022-11-23T02:14:29.5776182Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5776489Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5776880Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.5776900Z 
2022-11-23T02:14:29.5777156Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5777266Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5777285Z 
2022-11-23T02:14:29.5777393Z OK (skipped=1)
2022-11-23T02:14:29.5777412Z 
2022-11-23T02:14:29.5777533Z Generating XML reports...
2022-11-23T02:14:29.5778172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015345.xml
2022-11-23T02:14:29.5778550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5778794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5779173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5779358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5779378Z 
2022-11-23T02:14:29.5779485Z Running tests...
2022-11-23T02:14:29.5779743Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5780050Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5780318Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5780360Z 
2022-11-23T02:14:29.5780600Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5780710Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5780729Z 
2022-11-23T02:14:29.5780835Z OK (skipped=1)
2022-11-23T02:14:29.5780857Z 
2022-11-23T02:14:29.5780981Z Generating XML reports...
2022-11-23T02:14:29.5781422Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015347.xml
2022-11-23T02:14:29.5781781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5781955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5782326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5782493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5782599Z 
2022-11-23T02:14:29.5782694Z Running tests...
2022-11-23T02:14:29.5782955Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5783270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5783540Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5783560Z 
2022-11-23T02:14:29.5783816Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5783927Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5783946Z 
2022-11-23T02:14:29.5784053Z OK (skipped=1)
2022-11-23T02:14:29.5784072Z 
2022-11-23T02:14:29.5784194Z Generating XML reports...
2022-11-23T02:14:29.5784621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015349.xml
2022-11-23T02:14:29.5784986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5785160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5785537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5785726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5785745Z 
2022-11-23T02:14:29.5785852Z Running tests...
2022-11-23T02:14:29.5786112Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5786420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5786690Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5786710Z 
2022-11-23T02:14:29.5786950Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5787060Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5787082Z 
2022-11-23T02:14:29.5787188Z OK (skipped=1)
2022-11-23T02:14:29.5787207Z 
2022-11-23T02:14:29.5787330Z Generating XML reports...
2022-11-23T02:14:29.5787823Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015352.xml
2022-11-23T02:14:29.5788196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5788367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5788741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5788913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5788950Z 
2022-11-23T02:14:29.5789041Z Running tests...
2022-11-23T02:14:29.5789302Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5789614Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5789889Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5789908Z 
2022-11-23T02:14:29.5790165Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5790276Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5790295Z 
2022-11-23T02:14:29.5790402Z OK (skipped=1)
2022-11-23T02:14:29.5790421Z 
2022-11-23T02:14:29.5790545Z Generating XML reports...
2022-11-23T02:14:29.5790970Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015354.xml
2022-11-23T02:14:29.5791334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5791504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5791886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5792125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5792145Z 
2022-11-23T02:14:29.5792252Z Running tests...
2022-11-23T02:14:29.5792515Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5792864Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5793134Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.004s)
2022-11-23T02:14:29.5793154Z 
2022-11-23T02:14:29.5793394Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5793503Z Ran 1 test in 0.004s
2022-11-23T02:14:29.5793523Z 
2022-11-23T02:14:29.5793629Z OK (skipped=1)
2022-11-23T02:14:29.5793647Z 
2022-11-23T02:14:29.5793770Z Generating XML reports...
2022-11-23T02:14:29.5794211Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015357.xml
2022-11-23T02:14:29.5794582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5794758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5795132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5795319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5795339Z 
2022-11-23T02:14:29.5795429Z Running tests...
2022-11-23T02:14:29.5795685Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5795993Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5796256Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5796279Z 
2022-11-23T02:14:29.5796534Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5796644Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5796664Z 
2022-11-23T02:14:29.5796770Z OK (skipped=1)
2022-11-23T02:14:29.5796788Z 
2022-11-23T02:14:29.5796961Z Generating XML reports...
2022-11-23T02:14:29.5797389Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015359.xml
2022-11-23T02:14:29.5797753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5797925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5798292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5798480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5798503Z 
2022-11-23T02:14:29.5798613Z Running tests...
2022-11-23T02:14:29.5798871Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5799180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5799444Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5799464Z 
2022-11-23T02:14:29.5799702Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5799812Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5799831Z 
2022-11-23T02:14:29.5799937Z OK (skipped=1)
2022-11-23T02:14:29.5799955Z 
2022-11-23T02:14:29.5800078Z Generating XML reports...
2022-11-23T02:14:29.5800516Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015401.xml
2022-11-23T02:14:29.5800878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5801101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5801471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5801661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5801680Z 
2022-11-23T02:14:29.5801771Z Running tests...
2022-11-23T02:14:29.5802030Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5802340Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5802609Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5802628Z 
2022-11-23T02:14:29.5802887Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5802998Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5803021Z 
2022-11-23T02:14:29.5803128Z OK (skipped=1)
2022-11-23T02:14:29.5803147Z 
2022-11-23T02:14:29.5803270Z Generating XML reports...
2022-11-23T02:14:29.5803712Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015404.xml
2022-11-23T02:14:29.5804063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5804236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5804608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5804796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5804816Z 
2022-11-23T02:14:29.5804924Z Running tests...
2022-11-23T02:14:29.5805187Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5805498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5805763Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5805783Z 
2022-11-23T02:14:29.5806084Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5806183Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5806202Z 
2022-11-23T02:14:29.5806308Z OK (skipped=1)
2022-11-23T02:14:29.5806327Z 
2022-11-23T02:14:29.5806448Z Generating XML reports...
2022-11-23T02:14:29.5806890Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015406.xml
2022-11-23T02:14:29.5807254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5807426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5807794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5807983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5808003Z 
2022-11-23T02:14:29.5808093Z Running tests...
2022-11-23T02:14:29.5808352Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5808658Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5808914Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5808934Z 
2022-11-23T02:14:29.5809191Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5809302Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5809321Z 
2022-11-23T02:14:29.5809432Z OK (skipped=1)
2022-11-23T02:14:29.5809451Z 
2022-11-23T02:14:29.5809573Z Generating XML reports...
2022-11-23T02:14:29.5810010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015409.xml
2022-11-23T02:14:29.5810421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5810594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5810966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5811152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5811172Z 
2022-11-23T02:14:29.5811280Z Running tests...
2022-11-23T02:14:29.5811537Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5811845Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5812099Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5812123Z 
2022-11-23T02:14:29.5812382Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5812476Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5812495Z 
2022-11-23T02:14:29.5812600Z OK (skipped=1)
2022-11-23T02:14:29.5812619Z 
2022-11-23T02:14:29.5812742Z Generating XML reports...
2022-11-23T02:14:29.5813188Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015411.xml
2022-11-23T02:14:29.5813550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5813724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5814098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5814285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5814304Z 
2022-11-23T02:14:29.5814398Z Running tests...
2022-11-23T02:14:29.5814656Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5814959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5815279Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-11-23T02:14:29.5815301Z 
2022-11-23T02:14:29.5815559Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5815670Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5815689Z 
2022-11-23T02:14:29.5815795Z OK (skipped=1)
2022-11-23T02:14:29.5815814Z 
2022-11-23T02:14:29.5815936Z Generating XML reports...
2022-11-23T02:14:29.5816378Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015413.xml
2022-11-23T02:14:29.5816728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5816904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5817273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5817465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5817485Z 
2022-11-23T02:14:29.5817592Z Running tests...
2022-11-23T02:14:29.5817850Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5818346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5818641Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-11-23T02:14:29.5818656Z 
2022-11-23T02:14:29.5818910Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5819004Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5819023Z 
2022-11-23T02:14:29.5819206Z OK (skipped=1)
2022-11-23T02:14:29.5819226Z 
2022-11-23T02:14:29.5819348Z Generating XML reports...
2022-11-23T02:14:29.5819794Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015416.xml
2022-11-23T02:14:29.5820162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5820334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5820703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5820892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5820912Z 
2022-11-23T02:14:29.5821019Z Running tests...
2022-11-23T02:14:29.5821261Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5821567Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5821834Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5821854Z 
2022-11-23T02:14:29.5822113Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5822226Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5822245Z 
2022-11-23T02:14:29.5822351Z OK (skipped=1)
2022-11-23T02:14:29.5822369Z 
2022-11-23T02:14:29.5822492Z Generating XML reports...
2022-11-23T02:14:29.5822933Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015418.xml
2022-11-23T02:14:29.5823279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5823451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5823822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5824013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5824032Z 
2022-11-23T02:14:29.5824139Z Running tests...
2022-11-23T02:14:29.5824449Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5824765Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5825026Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5825242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25023
2022-11-23T02:14:29.5825440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25024
2022-11-23T02:14:29.5825805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5825976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5826349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5826535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5826891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5827060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5827424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5827591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5827834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5828071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5828521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5828913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5829144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5829367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5829468Z ok (6.864s)
2022-11-23T02:14:29.5829488Z 
2022-11-23T02:14:29.5829750Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5829844Z Ran 1 test in 6.865s
2022-11-23T02:14:29.5829863Z 
2022-11-23T02:14:29.5829955Z OK
2022-11-23T02:14:29.5829974Z 
2022-11-23T02:14:29.5830096Z Generating XML reports...
2022-11-23T02:14:29.5830541Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015420.xml
2022-11-23T02:14:29.5830912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5831085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5831463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5831651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5831670Z 
2022-11-23T02:14:29.5831778Z Running tests...
2022-11-23T02:14:29.5832021Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5832328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5832584Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5832604Z 
2022-11-23T02:14:29.5832858Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5832974Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5832993Z 
2022-11-23T02:14:29.5833101Z OK (skipped=1)
2022-11-23T02:14:29.5833119Z 
2022-11-23T02:14:29.5833242Z Generating XML reports...
2022-11-23T02:14:29.5833725Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015430.xml
2022-11-23T02:14:29.5834086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5834259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5834630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5834819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5834838Z 
2022-11-23T02:14:29.5834947Z Running tests...
2022-11-23T02:14:29.5835210Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5835520Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5835787Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5835806Z 
2022-11-23T02:14:29.5836063Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5836156Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5836175Z 
2022-11-23T02:14:29.5836282Z OK (skipped=1)
2022-11-23T02:14:29.5836301Z 
2022-11-23T02:14:29.5836424Z Generating XML reports...
2022-11-23T02:14:29.5836862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015432.xml
2022-11-23T02:14:29.5837226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5837397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5837816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5838003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5838026Z 
2022-11-23T02:14:29.5838133Z Running tests...
2022-11-23T02:14:29.5838375Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5838682Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5838945Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.5838964Z 
2022-11-23T02:14:29.5839222Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5839333Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5839353Z 
2022-11-23T02:14:29.5839460Z OK (skipped=1)
2022-11-23T02:14:29.5839479Z 
2022-11-23T02:14:29.5839606Z Generating XML reports...
2022-11-23T02:14:29.5840047Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015435.xml
2022-11-23T02:14:29.5840397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5840568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5840936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5841124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5841143Z 
2022-11-23T02:14:29.5841249Z Running tests...
2022-11-23T02:14:29.5841508Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5841817Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5842074Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5842289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25237
2022-11-23T02:14:29.5842485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25238
2022-11-23T02:14:29.5842896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5843074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5843447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5843634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5843992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5844164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5844531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5844699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5844944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5845185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5845580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5845973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5846200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5846424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5846821Z STAGE:2022-11-23 01:54:42 25238:25238 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5847142Z STAGE:2022-11-23 01:54:43 25237:25237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5847460Z STAGE:2022-11-23 01:54:43 25238:25238 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5847807Z STAGE:2022-11-23 01:54:43 25238:25238 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5848135Z STAGE:2022-11-23 01:54:43 25237:25237 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5848476Z STAGE:2022-11-23 01:54:43 25237:25237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5848800Z STAGE:2022-11-23 01:54:43 25238:25238 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5849112Z STAGE:2022-11-23 01:54:43 25237:25237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5849444Z STAGE:2022-11-23 01:54:43 25238:25238 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5849786Z STAGE:2022-11-23 01:54:43 25238:25238 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5850114Z STAGE:2022-11-23 01:54:43 25237:25237 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5850437Z STAGE:2022-11-23 01:54:43 25237:25237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5850759Z STAGE:2022-11-23 01:54:43 25238:25238 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5851079Z STAGE:2022-11-23 01:54:43 25237:25237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5851406Z STAGE:2022-11-23 01:54:43 25238:25238 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5851752Z STAGE:2022-11-23 01:54:43 25238:25238 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5852079Z STAGE:2022-11-23 01:54:43 25237:25237 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5852460Z STAGE:2022-11-23 01:54:43 25237:25237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5852568Z ok (6.328s)
2022-11-23T02:14:29.5852589Z 
2022-11-23T02:14:29.5852850Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5852945Z Ran 1 test in 6.328s
2022-11-23T02:14:29.5852964Z 
2022-11-23T02:14:29.5853055Z OK
2022-11-23T02:14:29.5853074Z 
2022-11-23T02:14:29.5853197Z Generating XML reports...
2022-11-23T02:14:29.5853643Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015437.xml
2022-11-23T02:14:29.5854009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5854185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5854566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5854757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5854776Z 
2022-11-23T02:14:29.5854885Z Running tests...
2022-11-23T02:14:29.5855130Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5855440Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5855706Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5855924Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25355
2022-11-23T02:14:29.5856140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25356
2022-11-23T02:14:29.5856594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5856766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5857144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5857311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5857667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5857831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5858369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5858557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5858797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5859039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5859435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5859826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5860036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5860257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5860585Z STAGE:2022-11-23 01:54:51 25356:25356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5860903Z STAGE:2022-11-23 01:54:51 25355:25355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5861234Z STAGE:2022-11-23 01:54:52 25356:25356 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5861563Z STAGE:2022-11-23 01:54:52 25355:25355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5861969Z STAGE:2022-11-23 01:54:52 25356:25356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5862313Z STAGE:2022-11-23 01:54:52 25355:25355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5862630Z STAGE:2022-11-23 01:54:52 25355:25355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5862929Z STAGE:2022-11-23 01:54:52 25356:25356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5863256Z STAGE:2022-11-23 01:54:52 25355:25355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5863591Z STAGE:2022-11-23 01:54:52 25355:25355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5863916Z STAGE:2022-11-23 01:54:52 25356:25356 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5864249Z STAGE:2022-11-23 01:54:52 25356:25356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5864570Z STAGE:2022-11-23 01:54:52 25355:25355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5864887Z STAGE:2022-11-23 01:54:52 25356:25356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5865212Z STAGE:2022-11-23 01:54:52 25355:25355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5865547Z STAGE:2022-11-23 01:54:52 25355:25355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5865860Z STAGE:2022-11-23 01:54:52 25356:25356 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5866194Z STAGE:2022-11-23 01:54:52 25356:25356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5866355Z ok (6.270s)
2022-11-23T02:14:29.5866376Z 
2022-11-23T02:14:29.5866637Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5866749Z Ran 1 test in 6.270s
2022-11-23T02:14:29.5866769Z 
2022-11-23T02:14:29.5866861Z OK
2022-11-23T02:14:29.5866880Z 
2022-11-23T02:14:29.5867003Z Generating XML reports...
2022-11-23T02:14:29.5867451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015446.xml
2022-11-23T02:14:29.5867805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5867976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5868348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5868534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5868558Z 
2022-11-23T02:14:29.5868664Z Running tests...
2022-11-23T02:14:29.5868921Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5869228Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5869494Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5869709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25473
2022-11-23T02:14:29.5869907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25474
2022-11-23T02:14:29.5870269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5870437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5870811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5871001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5871359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5871571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5871948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5872115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5872352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5872589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5872984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5873378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5873596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5873819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5874146Z STAGE:2022-11-23 01:55:00 25474:25474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5874464Z STAGE:2022-11-23 01:55:00 25473:25473 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5874777Z STAGE:2022-11-23 01:55:00 25474:25474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5875101Z STAGE:2022-11-23 01:55:00 25473:25473 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5875438Z STAGE:2022-11-23 01:55:00 25474:25474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5875837Z STAGE:2022-11-23 01:55:00 25473:25473 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5876157Z STAGE:2022-11-23 01:55:01 25474:25474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5876470Z STAGE:2022-11-23 01:55:01 25473:25473 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5876798Z STAGE:2022-11-23 01:55:01 25474:25474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5877133Z STAGE:2022-11-23 01:55:01 25474:25474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5877458Z STAGE:2022-11-23 01:55:01 25473:25473 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5877779Z STAGE:2022-11-23 01:55:01 25473:25473 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5878094Z STAGE:2022-11-23 01:55:01 25474:25474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5878414Z STAGE:2022-11-23 01:55:01 25473:25473 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5878743Z STAGE:2022-11-23 01:55:01 25474:25474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5879080Z STAGE:2022-11-23 01:55:01 25474:25474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5879403Z STAGE:2022-11-23 01:55:01 25473:25473 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5879739Z STAGE:2022-11-23 01:55:01 25473:25473 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5879840Z ok (6.384s)
2022-11-23T02:14:29.5879861Z 
2022-11-23T02:14:29.5880117Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5880212Z Ran 1 test in 6.384s
2022-11-23T02:14:29.5880231Z 
2022-11-23T02:14:29.5880326Z OK
2022-11-23T02:14:29.5880345Z 
2022-11-23T02:14:29.5880464Z Generating XML reports...
2022-11-23T02:14:29.5880914Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015455.xml
2022-11-23T02:14:29.5881322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5881502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5881880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5882066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5882086Z 
2022-11-23T02:14:29.5882194Z Running tests...
2022-11-23T02:14:29.5882437Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5882738Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5882981Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:14:29.5883000Z 
2022-11-23T02:14:29.5883255Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5883357Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5883379Z 
2022-11-23T02:14:29.5883480Z OK (skipped=1)
2022-11-23T02:14:29.5883499Z 
2022-11-23T02:14:29.5883615Z Generating XML reports...
2022-11-23T02:14:29.5884056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015504.xml
2022-11-23T02:14:29.5884406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5884579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5884949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5885184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5885204Z 
2022-11-23T02:14:29.5885312Z Running tests...
2022-11-23T02:14:29.5885566Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5885877Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5886130Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:14:29.5886149Z 
2022-11-23T02:14:29.5886407Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5886502Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5886521Z 
2022-11-23T02:14:29.5886624Z OK (skipped=1)
2022-11-23T02:14:29.5886643Z 
2022-11-23T02:14:29.5886759Z Generating XML reports...
2022-11-23T02:14:29.5887199Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015506.xml
2022-11-23T02:14:29.5887567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5887735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5888111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5888293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5888312Z 
2022-11-23T02:14:29.5888417Z Running tests...
2022-11-23T02:14:29.5888660Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5888961Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5889204Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5889416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25657
2022-11-23T02:14:29.5889626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25658
2022-11-23T02:14:29.5889988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5890200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5890577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5890747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5891103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5891268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5891627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5891811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5892048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5892281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5892714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5893105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5893315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5893533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5893627Z ok (6.982s)
2022-11-23T02:14:29.5893647Z 
2022-11-23T02:14:29.5893902Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5894059Z Ran 1 test in 6.982s
2022-11-23T02:14:29.5894078Z 
2022-11-23T02:14:29.5894165Z OK
2022-11-23T02:14:29.5894184Z 
2022-11-23T02:14:29.5894300Z Generating XML reports...
2022-11-23T02:14:29.5894743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015508.xml
2022-11-23T02:14:29.5895093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5895263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5895635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5895817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5895836Z 
2022-11-23T02:14:29.5895941Z Running tests...
2022-11-23T02:14:29.5896195Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5896506Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5896763Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5896975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25774
2022-11-23T02:14:29.5897172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25775
2022-11-23T02:14:29.5897529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5897695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5898276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5898462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5898823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5898995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5899424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5899598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5899834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5900070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5900465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5900852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5901079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5901299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5901396Z ok (6.857s)
2022-11-23T02:14:29.5901416Z 
2022-11-23T02:14:29.5901673Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5901768Z Ran 1 test in 6.857s
2022-11-23T02:14:29.5901788Z 
2022-11-23T02:14:29.5901875Z OK
2022-11-23T02:14:29.5901895Z 
2022-11-23T02:14:29.5902016Z Generating XML reports...
2022-11-23T02:14:29.5902456Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015518.xml
2022-11-23T02:14:29.5902819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5902988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5903356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5903604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5903624Z 
2022-11-23T02:14:29.5903729Z Running tests...
2022-11-23T02:14:29.5903978Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5904285Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5904540Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:14:29.5904559Z 
2022-11-23T02:14:29.5904808Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5904914Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5904934Z 
2022-11-23T02:14:29.5905036Z OK (skipped=1)
2022-11-23T02:14:29.5905055Z 
2022-11-23T02:14:29.5905173Z Generating XML reports...
2022-11-23T02:14:29.5905610Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015527.xml
2022-11-23T02:14:29.5905961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5906132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5906500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5906684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5906703Z 
2022-11-23T02:14:29.5906807Z Running tests...
2022-11-23T02:14:29.5907064Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5907369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5907628Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5907846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25924
2022-11-23T02:14:29.5908041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25925
2022-11-23T02:14:29.5908444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5908620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5908990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5909173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5909524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5909688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5910048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5910215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5910454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5910689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5911083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5911470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5911695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5911927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.5912212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5912447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.5912828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.5913216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.5913310Z ok (6.968s)
2022-11-23T02:14:29.5913329Z 
2022-11-23T02:14:29.5913584Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5913692Z Ran 1 test in 6.969s
2022-11-23T02:14:29.5913711Z 
2022-11-23T02:14:29.5913799Z OK
2022-11-23T02:14:29.5913819Z 
2022-11-23T02:14:29.5913939Z Generating XML reports...
2022-11-23T02:14:29.5914376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015530.xml
2022-11-23T02:14:29.5914742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5914898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5915273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5915458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5915477Z 
2022-11-23T02:14:29.5915585Z Running tests...
2022-11-23T02:14:29.5915842Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5916153Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5916402Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:14:29.5916422Z 
2022-11-23T02:14:29.5916682Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5916779Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5916814Z 
2022-11-23T02:14:29.5916904Z OK (skipped=1)
2022-11-23T02:14:29.5916923Z 
2022-11-23T02:14:29.5917046Z Generating XML reports...
2022-11-23T02:14:29.5917540Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015539.xml
2022-11-23T02:14:29.5917913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5918086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5918458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5918645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5918665Z 
2022-11-23T02:14:29.5918773Z Running tests...
2022-11-23T02:14:29.5919020Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5919329Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5919589Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5919807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26076
2022-11-23T02:14:29.5920020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26077
2022-11-23T02:14:29.5920384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5920558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5920931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5921117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5921511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5921678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5922048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5922296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5922580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5922859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5923295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5923723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5923938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5924201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5924549Z skip: Skipped due to small world size. (4.247s)
2022-11-23T02:14:29.5924571Z 
2022-11-23T02:14:29.5924826Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5924974Z Ran 1 test in 4.247s
2022-11-23T02:14:29.5924994Z 
2022-11-23T02:14:29.5925170Z OK (skipped=1)
2022-11-23T02:14:29.5925190Z 
2022-11-23T02:14:29.5925352Z Generating XML reports...
2022-11-23T02:14:29.5925836Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015542.xml
2022-11-23T02:14:29.5926238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5926464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5926875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5927152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5927217Z 
2022-11-23T02:14:29.5927318Z Running tests...
2022-11-23T02:14:29.5927624Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5927973Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5928297Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:14:29.5928318Z 
2022-11-23T02:14:29.5928613Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5928771Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5928791Z 
2022-11-23T02:14:29.5928933Z OK (skipped=1)
2022-11-23T02:14:29.5928957Z 
2022-11-23T02:14:29.5929114Z Generating XML reports...
2022-11-23T02:14:29.5929540Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015548.xml
2022-11-23T02:14:29.5929987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5930236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5930646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5930869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5930889Z 
2022-11-23T02:14:29.5931041Z Running tests...
2022-11-23T02:14:29.5931334Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5931681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5932062Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:14:29.5932083Z 
2022-11-23T02:14:29.5932326Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5932517Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5932536Z 
2022-11-23T02:14:29.5932683Z OK (skipped=1)
2022-11-23T02:14:29.5932702Z 
2022-11-23T02:14:29.5932872Z Generating XML reports...
2022-11-23T02:14:29.5933354Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015551.xml
2022-11-23T02:14:29.5933758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5933966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5934380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5934606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5934626Z 
2022-11-23T02:14:29.5934717Z Running tests...
2022-11-23T02:14:29.5935075Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5935440Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5935753Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5936534Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.668s)
2022-11-23T02:14:29.5936556Z 
2022-11-23T02:14:29.5936851Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5937004Z Ran 1 test in 1.668s
2022-11-23T02:14:29.5937024Z 
2022-11-23T02:14:29.5937167Z OK (skipped=1)
2022-11-23T02:14:29.5937186Z 
2022-11-23T02:14:29.5937346Z Generating XML reports...
2022-11-23T02:14:29.5938229Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015553.xml
2022-11-23T02:14:29.5938613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5938835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5939247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5939473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5939493Z 
2022-11-23T02:14:29.5939636Z Running tests...
2022-11-23T02:14:29.5939933Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5940288Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5940650Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5940855Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26281
2022-11-23T02:14:29.5941152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26282
2022-11-23T02:14:29.5941557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5941765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5942177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5942402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5964449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5964684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5965126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5965327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5965577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5965820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5966236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5966648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5966882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5967116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5967452Z STAGE:2022-11-23 01:56:03 26281:26281 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5967796Z STAGE:2022-11-23 01:56:03 26282:26282 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.5968137Z STAGE:2022-11-23 01:56:04 26281:26281 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5968485Z STAGE:2022-11-23 01:56:04 26282:26282 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.5968845Z STAGE:2022-11-23 01:56:04 26282:26282 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5969196Z STAGE:2022-11-23 01:56:04 26281:26281 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.5969304Z ok (7.822s)
2022-11-23T02:14:29.5969325Z 
2022-11-23T02:14:29.5969601Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5969713Z Ran 1 test in 7.823s
2022-11-23T02:14:29.5969733Z 
2022-11-23T02:14:29.5969807Z OK
2022-11-23T02:14:29.5969978Z 
2022-11-23T02:14:29.5970113Z Generating XML reports...
2022-11-23T02:14:29.5970594Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015557.xml
2022-11-23T02:14:29.5970983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5971156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5971554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5971749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5971774Z 
2022-11-23T02:14:29.5971875Z Running tests...
2022-11-23T02:14:29.5972154Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5972468Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5972777Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:14:29.5972797Z 
2022-11-23T02:14:29.5973062Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5973178Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5973197Z 
2022-11-23T02:14:29.5973302Z OK (skipped=1)
2022-11-23T02:14:29.5973321Z 
2022-11-23T02:14:29.5973437Z Generating XML reports...
2022-11-23T02:14:29.5973907Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015608.xml
2022-11-23T02:14:29.5974297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5974539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5974926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5975113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5975134Z 
2022-11-23T02:14:29.5975243Z Running tests...
2022-11-23T02:14:29.5975507Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5975836Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5976136Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5976920Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.659s)
2022-11-23T02:14:29.5976945Z 
2022-11-23T02:14:29.5977224Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5977337Z Ran 1 test in 1.659s
2022-11-23T02:14:29.5977356Z 
2022-11-23T02:14:29.5977449Z OK (skipped=1)
2022-11-23T02:14:29.5977474Z 
2022-11-23T02:14:29.5977583Z Generating XML reports...
2022-11-23T02:14:29.5978298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015610.xml
2022-11-23T02:14:29.5978691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5978861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5979229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5979418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5979438Z 
2022-11-23T02:14:29.5979536Z Running tests...
2022-11-23T02:14:29.5979882Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5980189Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5980480Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:14:29.5980500Z 
2022-11-23T02:14:29.5980754Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5980855Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5980874Z 
2022-11-23T02:14:29.5980979Z OK (skipped=1)
2022-11-23T02:14:29.5980998Z 
2022-11-23T02:14:29.5981110Z Generating XML reports...
2022-11-23T02:14:29.5981553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015614.xml
2022-11-23T02:14:29.5981925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5982098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5982460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5982637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5982656Z 
2022-11-23T02:14:29.5982761Z Running tests...
2022-11-23T02:14:29.5983022Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5983322Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5983604Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5983879Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26502
2022-11-23T02:14:29.5984094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26503
2022-11-23T02:14:29.5984450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5984621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5984992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5985169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5985524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5985691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5986047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5986234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5986466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.5986691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.5987086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5987480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.5987703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.5987917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.5988070Z skip: Skipped due to small world size. (4.386s)
2022-11-23T02:14:29.5988094Z 
2022-11-23T02:14:29.5988360Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5988461Z Ran 1 test in 4.386s
2022-11-23T02:14:29.5988480Z 
2022-11-23T02:14:29.5988585Z OK (skipped=1)
2022-11-23T02:14:29.5988604Z 
2022-11-23T02:14:29.5988761Z Generating XML reports...
2022-11-23T02:14:29.5989217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015617.xml
2022-11-23T02:14:29.5989573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5989746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5990118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5990302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5990325Z 
2022-11-23T02:14:29.5990422Z Running tests...
2022-11-23T02:14:29.5990683Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5990975Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5991260Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:14:29.5991279Z 
2022-11-23T02:14:29.5991527Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5991635Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5991654Z 
2022-11-23T02:14:29.5991757Z OK (skipped=1)
2022-11-23T02:14:29.5991776Z 
2022-11-23T02:14:29.5991888Z Generating XML reports...
2022-11-23T02:14:29.5992332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015624.xml
2022-11-23T02:14:29.5992736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5992960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5993317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5993510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5993529Z 
2022-11-23T02:14:29.5993637Z Running tests...
2022-11-23T02:14:29.5993893Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5994192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5994483Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:14:29.5994502Z 
2022-11-23T02:14:29.5994756Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5994860Z Ran 1 test in 0.002s
2022-11-23T02:14:29.5994879Z 
2022-11-23T02:14:29.5994984Z OK (skipped=1)
2022-11-23T02:14:29.5995003Z 
2022-11-23T02:14:29.5995108Z Generating XML reports...
2022-11-23T02:14:29.5995550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015626.xml
2022-11-23T02:14:29.5995902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5996075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5996441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5996623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5996642Z 
2022-11-23T02:14:29.5996738Z Running tests...
2022-11-23T02:14:29.5996993Z ----------------------------------------------------------------------
2022-11-23T02:14:29.5997292Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.5997555Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.5997816Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26673
2022-11-23T02:14:29.5998037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26674
2022-11-23T02:14:29.5998392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5998563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.5998938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.5999122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.5999469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.5999629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6000001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6000182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6000412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6000652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6001046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6001432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6001700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6001920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6002005Z ok (6.979s)
2022-11-23T02:14:29.6002025Z 
2022-11-23T02:14:29.6002288Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6002391Z Ran 1 test in 6.980s
2022-11-23T02:14:29.6002410Z 
2022-11-23T02:14:29.6002498Z OK
2022-11-23T02:14:29.6002517Z 
2022-11-23T02:14:29.6002637Z Generating XML reports...
2022-11-23T02:14:29.6003069Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015629.xml
2022-11-23T02:14:29.6003433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6003604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6003967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6004142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6004161Z 
2022-11-23T02:14:29.6004268Z Running tests...
2022-11-23T02:14:29.6004526Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6004828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6005117Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6005330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26790
2022-11-23T02:14:29.6005533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26791
2022-11-23T02:14:29.6005900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6006059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6006430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6006651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6007017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6007182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6007538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6007720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6007959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6008190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6008570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6008962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6009186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6009398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6009499Z ok (6.814s)
2022-11-23T02:14:29.6009519Z 
2022-11-23T02:14:29.6009777Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6009877Z Ran 1 test in 6.814s
2022-11-23T02:14:29.6009896Z 
2022-11-23T02:14:29.6009988Z OK
2022-11-23T02:14:29.6010007Z 
2022-11-23T02:14:29.6010114Z Generating XML reports...
2022-11-23T02:14:29.6010560Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015638.xml
2022-11-23T02:14:29.6010979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6011145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6011517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6011699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6011719Z 
2022-11-23T02:14:29.6011818Z Running tests...
2022-11-23T02:14:29.6012076Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6012385Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6012663Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:14:29.6012687Z 
2022-11-23T02:14:29.6012933Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6013042Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6013061Z 
2022-11-23T02:14:29.6013164Z OK (skipped=1)
2022-11-23T02:14:29.6013186Z 
2022-11-23T02:14:29.6013298Z Generating XML reports...
2022-11-23T02:14:29.6013743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015647.xml
2022-11-23T02:14:29.6014107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6014270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6014639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6014810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6014833Z 
2022-11-23T02:14:29.6014930Z Running tests...
2022-11-23T02:14:29.6015190Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6015497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6015828Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6016043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26940
2022-11-23T02:14:29.6016258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26941
2022-11-23T02:14:29.6016624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6016779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6017140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6017328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6017675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6017845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6018463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6018649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6018882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6019115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6019491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6019967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6020182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6020421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6020637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6020858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6021247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6021632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6021732Z ok (7.016s)
2022-11-23T02:14:29.6021756Z 
2022-11-23T02:14:29.6022000Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6022102Z Ran 1 test in 7.016s
2022-11-23T02:14:29.6022122Z 
2022-11-23T02:14:29.6022207Z OK
2022-11-23T02:14:29.6022226Z 
2022-11-23T02:14:29.6022337Z Generating XML reports...
2022-11-23T02:14:29.6022784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015650.xml
2022-11-23T02:14:29.6023146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6023308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6023681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6023849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6023884Z 
2022-11-23T02:14:29.6023979Z Running tests...
2022-11-23T02:14:29.6024230Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6024531Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6024878Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:14:29.6024900Z 
2022-11-23T02:14:29.6025163Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6025263Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6025282Z 
2022-11-23T02:14:29.6025391Z OK (skipped=1)
2022-11-23T02:14:29.6025410Z 
2022-11-23T02:14:29.6025527Z Generating XML reports...
2022-11-23T02:14:29.6026005Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015659.xml
2022-11-23T02:14:29.6026365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6026543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6026909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6027088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6027107Z 
2022-11-23T02:14:29.6027215Z Running tests...
2022-11-23T02:14:29.6027473Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6027773Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6028056Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6028255Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27092
2022-11-23T02:14:29.6028464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27093
2022-11-23T02:14:29.6028877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6029049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6029422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6029598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6029950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6030109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6030463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6030645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6030883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6031122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6031508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6031895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6032120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6032340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6032485Z skip: Skipped due to small world size. (4.154s)
2022-11-23T02:14:29.6032504Z 
2022-11-23T02:14:29.6032747Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6032856Z Ran 1 test in 4.155s
2022-11-23T02:14:29.6032880Z 
2022-11-23T02:14:29.6032976Z OK (skipped=1)
2022-11-23T02:14:29.6032995Z 
2022-11-23T02:14:29.6033114Z Generating XML reports...
2022-11-23T02:14:29.6033563Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015702.xml
2022-11-23T02:14:29.6033982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6034150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6034523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6034708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6034727Z 
2022-11-23T02:14:29.6034817Z Running tests...
2022-11-23T02:14:29.6035064Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6035368Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6035620Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6035837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27197
2022-11-23T02:14:29.6036049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27198
2022-11-23T02:14:29.6036403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6036571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6036925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6037107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6037452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6037670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6038036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6038224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6038466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6038706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6039100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6039474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6039702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6039930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6040165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6040403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6040797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6041181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6041282Z ok (7.135s)
2022-11-23T02:14:29.6041302Z 
2022-11-23T02:14:29.6041563Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6041656Z Ran 1 test in 7.136s
2022-11-23T02:14:29.6041675Z 
2022-11-23T02:14:29.6041768Z OK
2022-11-23T02:14:29.6041789Z 
2022-11-23T02:14:29.6041911Z Generating XML reports...
2022-11-23T02:14:29.6042356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015709.xml
2022-11-23T02:14:29.6042765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6042944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6043322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6043510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6043530Z 
2022-11-23T02:14:29.6043638Z Running tests...
2022-11-23T02:14:29.6043880Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6044187Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6044447Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6044664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27321
2022-11-23T02:14:29.6044879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27322
2022-11-23T02:14:29.6045244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6045415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6045789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6045958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6046317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6046490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6046953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6047135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6047380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6047625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6048022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6048410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6048618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6048842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6048994Z skip: Need at least 3 CUDA devices (4.210s)
2022-11-23T02:14:29.6049013Z 
2022-11-23T02:14:29.6049272Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6049381Z Ran 1 test in 4.210s
2022-11-23T02:14:29.6049403Z 
2022-11-23T02:14:29.6049515Z OK (skipped=1)
2022-11-23T02:14:29.6049534Z 
2022-11-23T02:14:29.6049656Z Generating XML reports...
2022-11-23T02:14:29.6050097Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015718.xml
2022-11-23T02:14:29.6050443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6050612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6050988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6051177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6051197Z 
2022-11-23T02:14:29.6051303Z Running tests...
2022-11-23T02:14:29.6051563Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6051919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6052171Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-11-23T02:14:29.6052191Z 
2022-11-23T02:14:29.6052451Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6052545Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6052564Z 
2022-11-23T02:14:29.6052672Z OK (skipped=1)
2022-11-23T02:14:29.6052691Z 
2022-11-23T02:14:29.6052814Z Generating XML reports...
2022-11-23T02:14:29.6053257Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015725.xml
2022-11-23T02:14:29.6053627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6053798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6054175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6054362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6054381Z 
2022-11-23T02:14:29.6054488Z Running tests...
2022-11-23T02:14:29.6054730Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6055039Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6055286Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T02:14:29.6055305Z 
2022-11-23T02:14:29.6055562Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6055724Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6055743Z 
2022-11-23T02:14:29.6055850Z OK (skipped=1)
2022-11-23T02:14:29.6055869Z 
2022-11-23T02:14:29.6055990Z Generating XML reports...
2022-11-23T02:14:29.6056437Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015727.xml
2022-11-23T02:14:29.6056784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6056956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6057324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6057511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6057530Z 
2022-11-23T02:14:29.6057635Z Running tests...
2022-11-23T02:14:29.6058105Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6058435Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6058681Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6058901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27492
2022-11-23T02:14:29.6059098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27493
2022-11-23T02:14:29.6059462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6059632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6060004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6060191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6060547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6060722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6061165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6061338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6061580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6061817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6062213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6062603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6062833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6063057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6063155Z ok (7.652s)
2022-11-23T02:14:29.6063175Z 
2022-11-23T02:14:29.6063436Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6063531Z Ran 1 test in 7.652s
2022-11-23T02:14:29.6063550Z 
2022-11-23T02:14:29.6063637Z OK
2022-11-23T02:14:29.6063656Z 
2022-11-23T02:14:29.6063775Z Generating XML reports...
2022-11-23T02:14:29.6064220Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015730.xml
2022-11-23T02:14:29.6064583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6064755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6065194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6065381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6065400Z 
2022-11-23T02:14:29.6065508Z Running tests...
2022-11-23T02:14:29.6065752Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6066060Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6066321Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T02:14:29.6066341Z 
2022-11-23T02:14:29.6066596Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6066710Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6066729Z 
2022-11-23T02:14:29.6066835Z OK (skipped=1)
2022-11-23T02:14:29.6066854Z 
2022-11-23T02:14:29.6066975Z Generating XML reports...
2022-11-23T02:14:29.6067419Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015740.xml
2022-11-23T02:14:29.6067762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6067940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6068309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6068492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6068510Z 
2022-11-23T02:14:29.6068617Z Running tests...
2022-11-23T02:14:29.6068872Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6069182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6069446Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6069665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27640
2022-11-23T02:14:29.6069863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27641
2022-11-23T02:14:29.6070269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6070446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6070824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6071008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6071364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6071536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6071900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6072074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6072313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6072552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6072947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6073338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6073564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6073787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6073991Z skip: Skipped due to small world size. (4.204s)
2022-11-23T02:14:29.6074011Z 
2022-11-23T02:14:29.6074273Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6074366Z Ran 1 test in 4.204s
2022-11-23T02:14:29.6074402Z 
2022-11-23T02:14:29.6074492Z OK (skipped=1)
2022-11-23T02:14:29.6074514Z 
2022-11-23T02:14:29.6074634Z Generating XML reports...
2022-11-23T02:14:29.6075080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015742.xml
2022-11-23T02:14:29.6075446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6075618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6075991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6076177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6076200Z 
2022-11-23T02:14:29.6076307Z Running tests...
2022-11-23T02:14:29.6076550Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6076860Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6077114Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T02:14:29.6077134Z 
2022-11-23T02:14:29.6077388Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6077494Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6077513Z 
2022-11-23T02:14:29.6077620Z OK (skipped=1)
2022-11-23T02:14:29.6077639Z 
2022-11-23T02:14:29.6077761Z Generating XML reports...
2022-11-23T02:14:29.6078201Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015749.xml
2022-11-23T02:14:29.6078564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6078723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6079100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6079333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6079354Z 
2022-11-23T02:14:29.6079466Z Running tests...
2022-11-23T02:14:29.6079727Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6080037Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6080294Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6080508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27778
2022-11-23T02:14:29.6080705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27779
2022-11-23T02:14:29.6081079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6081251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6081627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6081813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6082171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6082344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6082710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6082893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6083169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6083409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6083809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6084201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6084425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6084649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6084807Z skip: Skipped due to small world size. (4.237s)
2022-11-23T02:14:29.6084827Z 
2022-11-23T02:14:29.6085089Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6085187Z Ran 1 test in 4.238s
2022-11-23T02:14:29.6085222Z 
2022-11-23T02:14:29.6085311Z OK (skipped=1)
2022-11-23T02:14:29.6085330Z 
2022-11-23T02:14:29.6085453Z Generating XML reports...
2022-11-23T02:14:29.6085903Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015751.xml
2022-11-23T02:14:29.6086266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6086437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6086811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6086999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6087019Z 
2022-11-23T02:14:29.6087126Z Running tests...
2022-11-23T02:14:29.6087369Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6087683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6087958Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T02:14:29.6087977Z 
2022-11-23T02:14:29.6088272Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6088389Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6088408Z 
2022-11-23T02:14:29.6088516Z OK (skipped=1)
2022-11-23T02:14:29.6088534Z 
2022-11-23T02:14:29.6088656Z Generating XML reports...
2022-11-23T02:14:29.6089100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015758.xml
2022-11-23T02:14:29.6089466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6089620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6089991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6090182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6090201Z 
2022-11-23T02:14:29.6090309Z Running tests...
2022-11-23T02:14:29.6090569Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6090877Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6091147Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T02:14:29.6091167Z 
2022-11-23T02:14:29.6091424Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6091535Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6091554Z 
2022-11-23T02:14:29.6091644Z OK (skipped=1)
2022-11-23T02:14:29.6091663Z 
2022-11-23T02:14:29.6091786Z Generating XML reports...
2022-11-23T02:14:29.6092282Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015801.xml
2022-11-23T02:14:29.6092645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6092866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6093239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6093428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6093447Z 
2022-11-23T02:14:29.6093553Z Running tests...
2022-11-23T02:14:29.6093796Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6094109Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6094378Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T02:14:29.6094401Z 
2022-11-23T02:14:29.6094659Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6094770Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6094789Z 
2022-11-23T02:14:29.6094896Z OK (skipped=1)
2022-11-23T02:14:29.6094914Z 
2022-11-23T02:14:29.6095040Z Generating XML reports...
2022-11-23T02:14:29.6095478Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015803.xml
2022-11-23T02:14:29.6095847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6096001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6096375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6096560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6096584Z 
2022-11-23T02:14:29.6096688Z Running tests...
2022-11-23T02:14:29.6096944Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6097251Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6097559Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-11-23T02:14:29.6097580Z 
2022-11-23T02:14:29.6097842Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6098160Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6098186Z 
2022-11-23T02:14:29.6098275Z OK (skipped=1)
2022-11-23T02:14:29.6098296Z 
2022-11-23T02:14:29.6098423Z Generating XML reports...
2022-11-23T02:14:29.6098872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015805.xml
2022-11-23T02:14:29.6099234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6099411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6099786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6099979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6099998Z 
2022-11-23T02:14:29.6100107Z Running tests...
2022-11-23T02:14:29.6100348Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6100659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6100914Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-11-23T02:14:29.6100933Z 
2022-11-23T02:14:29.6101188Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6101297Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6101386Z 
2022-11-23T02:14:29.6101499Z OK (skipped=1)
2022-11-23T02:14:29.6101518Z 
2022-11-23T02:14:29.6101640Z Generating XML reports...
2022-11-23T02:14:29.6102084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015808.xml
2022-11-23T02:14:29.6102450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6102605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6102974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6103160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6103179Z 
2022-11-23T02:14:29.6103287Z Running tests...
2022-11-23T02:14:29.6103543Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6103854Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6104135Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6104348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28048
2022-11-23T02:14:29.6104564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28049
2022-11-23T02:14:29.6104912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6105088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6105460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6105643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6105996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6106170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6106537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6106775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6107008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6107247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6107644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6108032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6108257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6108485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6108718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6108960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6109356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6109726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6109963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:14:29.6110197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:14:29.6110591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6111030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6111137Z ok (4.456s)
2022-11-23T02:14:29.6111159Z 
2022-11-23T02:14:29.6111420Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6111531Z Ran 1 test in 4.456s
2022-11-23T02:14:29.6111550Z 
2022-11-23T02:14:29.6111643Z OK
2022-11-23T02:14:29.6111662Z 
2022-11-23T02:14:29.6111768Z Generating XML reports...
2022-11-23T02:14:29.6112211Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015810.xml
2022-11-23T02:14:29.6112573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6112745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6113123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6113311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6113330Z 
2022-11-23T02:14:29.6113440Z Running tests...
2022-11-23T02:14:29.6113699Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6114008Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6114252Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6114471Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28161
2022-11-23T02:14:29.6114684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28162
2022-11-23T02:14:29.6115049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6115224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6115598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6115833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6116203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6116357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6116726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6116911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6117153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6117393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6117792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6118184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6118412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6118637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6118722Z ok (5.843s)
2022-11-23T02:14:29.6118742Z 
2022-11-23T02:14:29.6119000Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6119111Z Ran 1 test in 5.844s
2022-11-23T02:14:29.6119130Z 
2022-11-23T02:14:29.6119222Z OK
2022-11-23T02:14:29.6119241Z 
2022-11-23T02:14:29.6119362Z Generating XML reports...
2022-11-23T02:14:29.6119857Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015817.xml
2022-11-23T02:14:29.6120218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6120394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6120771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6120942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6120962Z 
2022-11-23T02:14:29.6121070Z Running tests...
2022-11-23T02:14:29.6121330Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6121638Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6121914Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6122134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28277
2022-11-23T02:14:29.6122347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28278
2022-11-23T02:14:29.6122715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6122868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6123241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6123426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6123781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6123952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6124326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6124508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6124794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6125039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6125423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6125815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6126040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6126266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6126428Z skip: Skipped due to small world size. (4.248s)
2022-11-23T02:14:29.6126448Z 
2022-11-23T02:14:29.6126709Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6126821Z Ran 1 test in 4.248s
2022-11-23T02:14:29.6126840Z 
2022-11-23T02:14:29.6126948Z OK (skipped=1)
2022-11-23T02:14:29.6126967Z 
2022-11-23T02:14:29.6127072Z Generating XML reports...
2022-11-23T02:14:29.6127514Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015826.xml
2022-11-23T02:14:29.6127883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6128056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6128426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6128613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6128679Z 
2022-11-23T02:14:29.6128791Z Running tests...
2022-11-23T02:14:29.6129048Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6129362Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6129609Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6129823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28382
2022-11-23T02:14:29.6130037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28383
2022-11-23T02:14:29.6130406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6130577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6130946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6131140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6131499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6131653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6132018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6132201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6132442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6132680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6133072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6133470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6133695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6133963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6134053Z ok (5.622s)
2022-11-23T02:14:29.6134090Z 
2022-11-23T02:14:29.6134332Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6134445Z Ran 1 test in 5.622s
2022-11-23T02:14:29.6134464Z 
2022-11-23T02:14:29.6134555Z OK
2022-11-23T02:14:29.6134574Z 
2022-11-23T02:14:29.6134696Z Generating XML reports...
2022-11-23T02:14:29.6135138Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015832.xml
2022-11-23T02:14:29.6135501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6135676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6136049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6136222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6136241Z 
2022-11-23T02:14:29.6136346Z Running tests...
2022-11-23T02:14:29.6136604Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6136911Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6137175Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6137389Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28488
2022-11-23T02:14:29.6137603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28489
2022-11-23T02:14:29.6138336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6138494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6138877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6139062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6139419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6139590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6139959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6140148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6140391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6140630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6141014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6141403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6141632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6141855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6141957Z ok (4.374s)
2022-11-23T02:14:29.6141977Z 
2022-11-23T02:14:29.6142237Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6142347Z Ran 1 test in 4.374s
2022-11-23T02:14:29.6142370Z 
2022-11-23T02:14:29.6142460Z OK
2022-11-23T02:14:29.6142479Z 
2022-11-23T02:14:29.6142586Z Generating XML reports...
2022-11-23T02:14:29.6143033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015841.xml
2022-11-23T02:14:29.6143470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6143653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6144028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6144216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6144235Z 
2022-11-23T02:14:29.6144344Z Running tests...
2022-11-23T02:14:29.6144604Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6144913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6145180Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6145394Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28593
2022-11-23T02:14:29.6145608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28594
2022-11-23T02:14:29.6145970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6146141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6146514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6146702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6147062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6147298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6147651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6147840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6148083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6148319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6148716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6149108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6149335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6149561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6149646Z ok (5.880s)
2022-11-23T02:14:29.6149681Z 
2022-11-23T02:14:29.6149929Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6150040Z Ran 1 test in 5.881s
2022-11-23T02:14:29.6150059Z 
2022-11-23T02:14:29.6150150Z OK
2022-11-23T02:14:29.6150169Z 
2022-11-23T02:14:29.6150292Z Generating XML reports...
2022-11-23T02:14:29.6150735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015847.xml
2022-11-23T02:14:29.6151102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6151276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6151651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6151826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6151845Z 
2022-11-23T02:14:29.6151952Z Running tests...
2022-11-23T02:14:29.6152257Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6152575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6152842Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6153059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28709
2022-11-23T02:14:29.6153272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28710
2022-11-23T02:14:29.6153639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6153794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6154174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6154358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6154720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6154890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6155260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6155444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6155683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6155923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6156355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6156739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6156969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6157193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6157295Z ok (6.132s)
2022-11-23T02:14:29.6157315Z 
2022-11-23T02:14:29.6157573Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6157685Z Ran 1 test in 6.132s
2022-11-23T02:14:29.6157704Z 
2022-11-23T02:14:29.6157800Z OK
2022-11-23T02:14:29.6157819Z 
2022-11-23T02:14:29.6157924Z Generating XML reports...
2022-11-23T02:14:29.6158370Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015856.xml
2022-11-23T02:14:29.6158740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6158913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6159291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6159477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6159496Z 
2022-11-23T02:14:29.6159604Z Running tests...
2022-11-23T02:14:29.6159862Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6160174Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6160426Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6160637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28823
2022-11-23T02:14:29.6160856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28824
2022-11-23T02:14:29.6161222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6161451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6161832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6162019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6162376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6162546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6162901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6163090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6163329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6163574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6163968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6164358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6164585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6164808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6164893Z ok (4.364s)
2022-11-23T02:14:29.6164929Z 
2022-11-23T02:14:29.6165170Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6165326Z Ran 1 test in 4.364s
2022-11-23T02:14:29.6165346Z 
2022-11-23T02:14:29.6165438Z OK
2022-11-23T02:14:29.6165457Z 
2022-11-23T02:14:29.6165579Z Generating XML reports...
2022-11-23T02:14:29.6166032Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015905.xml
2022-11-23T02:14:29.6166395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6166566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6166937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6167107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6167126Z 
2022-11-23T02:14:29.6167234Z Running tests...
2022-11-23T02:14:29.6167493Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6167808Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6168056Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.6168079Z 
2022-11-23T02:14:29.6168339Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6168450Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6168469Z 
2022-11-23T02:14:29.6168577Z OK (skipped=1)
2022-11-23T02:14:29.6168596Z 
2022-11-23T02:14:29.6168718Z Generating XML reports...
2022-11-23T02:14:29.6169143Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015912.xml
2022-11-23T02:14:29.6169506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6169677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6170055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6170241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6170260Z 
2022-11-23T02:14:29.6170415Z Running tests...
2022-11-23T02:14:29.6170683Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6170993Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6171226Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6171970Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.620s)
2022-11-23T02:14:29.6171995Z 
2022-11-23T02:14:29.6172255Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6172368Z Ran 1 test in 1.620s
2022-11-23T02:14:29.6172387Z 
2022-11-23T02:14:29.6172494Z OK (skipped=1)
2022-11-23T02:14:29.6172513Z 
2022-11-23T02:14:29.6172640Z Generating XML reports...
2022-11-23T02:14:29.6173086Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015914.xml
2022-11-23T02:14:29.6173452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6173624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6173998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6174168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6174235Z 
2022-11-23T02:14:29.6174346Z Running tests...
2022-11-23T02:14:29.6174606Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6174915Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6175181Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.6175200Z 
2022-11-23T02:14:29.6175459Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6175569Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6175587Z 
2022-11-23T02:14:29.6175695Z OK (skipped=1)
2022-11-23T02:14:29.6175713Z 
2022-11-23T02:14:29.6175834Z Generating XML reports...
2022-11-23T02:14:29.6176258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015918.xml
2022-11-23T02:14:29.6176622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6176797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6177169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6177359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6177378Z 
2022-11-23T02:14:29.6177486Z Running tests...
2022-11-23T02:14:29.6177745Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6178274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6178521Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.6178558Z 
2022-11-23T02:14:29.6178799Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6178911Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6178930Z 
2022-11-23T02:14:29.6179045Z OK (skipped=1)
2022-11-23T02:14:29.6179064Z 
2022-11-23T02:14:29.6179187Z Generating XML reports...
2022-11-23T02:14:29.6179628Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015920.xml
2022-11-23T02:14:29.6180067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6180251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6180626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6180798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6180818Z 
2022-11-23T02:14:29.6180927Z Running tests...
2022-11-23T02:14:29.6181188Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6181497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6181763Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-11-23T02:14:29.6181783Z 
2022-11-23T02:14:29.6182038Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6182151Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6182170Z 
2022-11-23T02:14:29.6182277Z OK (skipped=1)
2022-11-23T02:14:29.6182295Z 
2022-11-23T02:14:29.6182415Z Generating XML reports...
2022-11-23T02:14:29.6182841Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015923.xml
2022-11-23T02:14:29.6183202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6183374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6183748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6183998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6184017Z 
2022-11-23T02:14:29.6184125Z Running tests...
2022-11-23T02:14:29.6184387Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6184702Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6184944Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6185684Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.701s)
2022-11-23T02:14:29.6185723Z 
2022-11-23T02:14:29.6185964Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6186080Z Ran 1 test in 1.701s
2022-11-23T02:14:29.6186100Z 
2022-11-23T02:14:29.6186206Z OK (skipped=1)
2022-11-23T02:14:29.6186224Z 
2022-11-23T02:14:29.6186346Z Generating XML reports...
2022-11-23T02:14:29.6186787Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015925.xml
2022-11-23T02:14:29.6187154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6187329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6187700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6187870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6187889Z 
2022-11-23T02:14:29.6187996Z Running tests...
2022-11-23T02:14:29.6188262Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6188576Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6188883Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6189692Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.607s)
2022-11-23T02:14:29.6189714Z 
2022-11-23T02:14:29.6189979Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6190090Z Ran 1 test in 1.608s
2022-11-23T02:14:29.6190109Z 
2022-11-23T02:14:29.6190215Z OK (skipped=1)
2022-11-23T02:14:29.6190234Z 
2022-11-23T02:14:29.6190356Z Generating XML reports...
2022-11-23T02:14:29.6190780Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015929.xml
2022-11-23T02:14:29.6191152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6191327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6191701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6191886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6191906Z 
2022-11-23T02:14:29.6192013Z Running tests...
2022-11-23T02:14:29.6192273Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6192583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6192914Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6193707Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.651s)
2022-11-23T02:14:29.6193746Z 
2022-11-23T02:14:29.6193985Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6194096Z Ran 1 test in 1.651s
2022-11-23T02:14:29.6194116Z 
2022-11-23T02:14:29.6194221Z OK (skipped=1)
2022-11-23T02:14:29.6194240Z 
2022-11-23T02:14:29.6194361Z Generating XML reports...
2022-11-23T02:14:29.6194801Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015933.xml
2022-11-23T02:14:29.6195166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6195341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6195715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6195886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6195923Z 
2022-11-23T02:14:29.6196013Z Running tests...
2022-11-23T02:14:29.6196273Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6196582Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6196844Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6197061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29196
2022-11-23T02:14:29.6197275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29197
2022-11-23T02:14:29.6197644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6197799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6198217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6198409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6198771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6198941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6199309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6199494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6199734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6199977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6200360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6200755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6200982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6201203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6201455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83ha4wwh
2022-11-23T02:14:29.6201723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83ha4wwh/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6201974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiqini5s
2022-11-23T02:14:29.6202288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiqini5s/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6202523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6202736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6202840Z ok (6.362s)
2022-11-23T02:14:29.6202859Z 
2022-11-23T02:14:29.6203124Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6203234Z Ran 1 test in 6.362s
2022-11-23T02:14:29.6203254Z 
2022-11-23T02:14:29.6203344Z OK
2022-11-23T02:14:29.6203363Z 
2022-11-23T02:14:29.6203485Z Generating XML reports...
2022-11-23T02:14:29.6203929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015938.xml
2022-11-23T02:14:29.6204293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6204453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6204832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6205019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6205038Z 
2022-11-23T02:14:29.6205145Z Running tests...
2022-11-23T02:14:29.6205410Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6205719Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6205988Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6206206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29314
2022-11-23T02:14:29.6206421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29315
2022-11-23T02:14:29.6206773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6206945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6207363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6207556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6207915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6208086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6208455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6208641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6208869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6209108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6209503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6209891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6210115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6210338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6210588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ux4nhym
2022-11-23T02:14:29.6210854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ux4nhym/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6211153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmyxrhhyn
2022-11-23T02:14:29.6211402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmyxrhhyn/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6211638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6211868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6212100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6212328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6212429Z ok (6.322s)
2022-11-23T02:14:29.6212449Z 
2022-11-23T02:14:29.6212715Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6212826Z Ran 1 test in 6.322s
2022-11-23T02:14:29.6212850Z 
2022-11-23T02:14:29.6212925Z OK
2022-11-23T02:14:29.6212960Z 
2022-11-23T02:14:29.6213065Z Generating XML reports...
2022-11-23T02:14:29.6213512Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015946.xml
2022-11-23T02:14:29.6213881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6214053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6214429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6214616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6214634Z 
2022-11-23T02:14:29.6214745Z Running tests...
2022-11-23T02:14:29.6215002Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6215293Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6215565Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6216348Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.606s)
2022-11-23T02:14:29.6216371Z 
2022-11-23T02:14:29.6216637Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6216749Z Ran 1 test in 1.607s
2022-11-23T02:14:29.6216768Z 
2022-11-23T02:14:29.6216875Z OK (skipped=1)
2022-11-23T02:14:29.6216894Z 
2022-11-23T02:14:29.6217017Z Generating XML reports...
2022-11-23T02:14:29.6217462Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015955.xml
2022-11-23T02:14:29.6217835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6218231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6218608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6218792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6218811Z 
2022-11-23T02:14:29.6218919Z Running tests...
2022-11-23T02:14:29.6219180Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6219495Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6219780Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6220513Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2022-11-23T02:14:29.6220606Z 
2022-11-23T02:14:29.6220878Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6220992Z Ran 1 test in 1.634s
2022-11-23T02:14:29.6221011Z 
2022-11-23T02:14:29.6221101Z OK (skipped=1)
2022-11-23T02:14:29.6221135Z 
2022-11-23T02:14:29.6221241Z Generating XML reports...
2022-11-23T02:14:29.6221684Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015959.xml
2022-11-23T02:14:29.6222049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6222221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6222596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6222786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6222805Z 
2022-11-23T02:14:29.6222911Z Running tests...
2022-11-23T02:14:29.6223172Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6223467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6223746Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6223961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29500
2022-11-23T02:14:29.6224173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29501
2022-11-23T02:14:29.6224539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6224715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6225086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6225343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6225717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6225869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6226236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6226423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6226666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6226908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6227307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6227704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6227930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6228137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6228390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_apiwf_b
2022-11-23T02:14:29.6228654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_apiwf_b/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6228905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpro5ud4ro
2022-11-23T02:14:29.6229166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpro5ud4ro/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6229426Z <class 'torch.nn.parameter.Parameter'>
2022-11-23T02:14:29.6229530Z ok (6.748s)
2022-11-23T02:14:29.6229550Z 
2022-11-23T02:14:29.6229811Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6229923Z Ran 1 test in 6.749s
2022-11-23T02:14:29.6229943Z 
2022-11-23T02:14:29.6230018Z OK
2022-11-23T02:14:29.6230036Z 
2022-11-23T02:14:29.6230160Z Generating XML reports...
2022-11-23T02:14:29.6230608Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020004.xml
2022-11-23T02:14:29.6230973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6231145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6231516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6231706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6231726Z 
2022-11-23T02:14:29.6231834Z Running tests...
2022-11-23T02:14:29.6232075Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6232382Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6232683Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6232900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29615
2022-11-23T02:14:29.6233110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29616
2022-11-23T02:14:29.6233476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6233648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6234027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6234213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6234600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6234779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6235151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6235336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6235578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6235819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6236218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6236608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6236835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6237042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6237294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7k8_dq4
2022-11-23T02:14:29.6237554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7k8_dq4/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6237805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxs6xon8h
2022-11-23T02:14:29.6238071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxs6xon8h/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6238221Z ok (6.843s)
2022-11-23T02:14:29.6238240Z 
2022-11-23T02:14:29.6238504Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6238617Z Ran 1 test in 6.843s
2022-11-23T02:14:29.6238637Z 
2022-11-23T02:14:29.6238711Z OK
2022-11-23T02:14:29.6238733Z 
2022-11-23T02:14:29.6238855Z Generating XML reports...
2022-11-23T02:14:29.6239299Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020013.xml
2022-11-23T02:14:29.6239662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6239833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6240205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6240393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6240416Z 
2022-11-23T02:14:29.6240523Z Running tests...
2022-11-23T02:14:29.6240783Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6241077Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6241336Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6241550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29730
2022-11-23T02:14:29.6241762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29731
2022-11-23T02:14:29.6242126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6242295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6242667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6242859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6243199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6243415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6243794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6243978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6244219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6244461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6244858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6245254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6245479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6245689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6245942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrabe8cs
2022-11-23T02:14:29.6246211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrabe8cs/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6246462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzw2m1lf
2022-11-23T02:14:29.6246729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzw2m1lf/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6246961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6247244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6247349Z ok (7.458s)
2022-11-23T02:14:29.6247368Z 
2022-11-23T02:14:29.6247616Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6247731Z Ran 1 test in 7.458s
2022-11-23T02:14:29.6247750Z 
2022-11-23T02:14:29.6247841Z OK
2022-11-23T02:14:29.6247861Z 
2022-11-23T02:14:29.6247984Z Generating XML reports...
2022-11-23T02:14:29.6248431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020022.xml
2022-11-23T02:14:29.6248799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6248975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6249351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6249543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6249563Z 
2022-11-23T02:14:29.6249653Z Running tests...
2022-11-23T02:14:29.6249917Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6250226Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6250511Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6250725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29849
2022-11-23T02:14:29.6250939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29850
2022-11-23T02:14:29.6251306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6251478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6251839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6252028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6252431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6252608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6252973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6253157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6253400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6253639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6254034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6254415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6254643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6254863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6255115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9a5iuwn
2022-11-23T02:14:29.6255380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9a5iuwn/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6255628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3762acy
2022-11-23T02:14:29.6255887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3762acy/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6256713Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:14:29.6256819Z ok (6.368s)
2022-11-23T02:14:29.6256839Z 
2022-11-23T02:14:29.6257103Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6257198Z Ran 1 test in 6.368s
2022-11-23T02:14:29.6257236Z 
2022-11-23T02:14:29.6257311Z OK
2022-11-23T02:14:29.6257329Z 
2022-11-23T02:14:29.6257453Z Generating XML reports...
2022-11-23T02:14:29.6258106Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020032.xml
2022-11-23T02:14:29.6258498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6258664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6259041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6259229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6259248Z 
2022-11-23T02:14:29.6259356Z Running tests...
2022-11-23T02:14:29.6259594Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6259904Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6260181Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6260921Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2022-11-23T02:14:29.6260945Z 
2022-11-23T02:14:29.6261276Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6261394Z Ran 1 test in 1.630s
2022-11-23T02:14:29.6261414Z 
2022-11-23T02:14:29.6261522Z OK (skipped=1)
2022-11-23T02:14:29.6261541Z 
2022-11-23T02:14:29.6261663Z Generating XML reports...
2022-11-23T02:14:29.6262108Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020041.xml
2022-11-23T02:14:29.6262472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6262630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6263005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6263192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6263212Z 
2022-11-23T02:14:29.6263318Z Running tests...
2022-11-23T02:14:29.6263582Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6263891Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6264214Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-11-23T02:14:29.6264233Z 
2022-11-23T02:14:29.6264487Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6264580Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6264617Z 
2022-11-23T02:14:29.6264706Z OK (skipped=1)
2022-11-23T02:14:29.6264725Z 
2022-11-23T02:14:29.6264847Z Generating XML reports...
2022-11-23T02:14:29.6265363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020046.xml
2022-11-23T02:14:29.6265729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6265905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6266280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6266468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6266487Z 
2022-11-23T02:14:29.6266594Z Running tests...
2022-11-23T02:14:29.6266835Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6267148Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6267390Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6268137Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.668s)
2022-11-23T02:14:29.6268158Z 
2022-11-23T02:14:29.6268417Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6268527Z Ran 1 test in 1.668s
2022-11-23T02:14:29.6268546Z 
2022-11-23T02:14:29.6268653Z OK (skipped=1)
2022-11-23T02:14:29.6268671Z 
2022-11-23T02:14:29.6268796Z Generating XML reports...
2022-11-23T02:14:29.6269242Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020048.xml
2022-11-23T02:14:29.6269605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6269766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6270142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6270329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6270394Z 
2022-11-23T02:14:29.6270507Z Running tests...
2022-11-23T02:14:29.6270770Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6271080Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6271355Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6271574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30068
2022-11-23T02:14:29.6271772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30069
2022-11-23T02:14:29.6272137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6272316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6272692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6272878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6273234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6273402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6273769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6273955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6274180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6274471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6274872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6275263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6275491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6275719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6275973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7h2uwz_
2022-11-23T02:14:29.6276240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7h2uwz_/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6276491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb947bfiv
2022-11-23T02:14:29.6276742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb947bfiv/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6277533Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T02:14:29.6277863Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T02:14:29.6278641Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T02:14:29.6278970Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T02:14:29.6279072Z ok (7.564s)
2022-11-23T02:14:29.6279092Z 
2022-11-23T02:14:29.6279351Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6279507Z Ran 1 test in 7.564s
2022-11-23T02:14:29.6279528Z 
2022-11-23T02:14:29.6279623Z OK
2022-11-23T02:14:29.6279641Z 
2022-11-23T02:14:29.6279766Z Generating XML reports...
2022-11-23T02:14:29.6280196Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020052.xml
2022-11-23T02:14:29.6280559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6280731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6281102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6281295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6281314Z 
2022-11-23T02:14:29.6281420Z Running tests...
2022-11-23T02:14:29.6281678Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6281990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6282255Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6282974Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-11-23T02:14:29.6283012Z 
2022-11-23T02:14:29.6283253Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6283415Z Ran 1 test in 1.641s
2022-11-23T02:14:29.6283435Z 
2022-11-23T02:14:29.6283542Z OK (skipped=1)
2022-11-23T02:14:29.6283560Z 
2022-11-23T02:14:29.6283683Z Generating XML reports...
2022-11-23T02:14:29.6284129Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020102.xml
2022-11-23T02:14:29.6284494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6284670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6285043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6285212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6285248Z 
2022-11-23T02:14:29.6285338Z Running tests...
2022-11-23T02:14:29.6285598Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6285912Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6286178Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6286918Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.656s)
2022-11-23T02:14:29.6286939Z 
2022-11-23T02:14:29.6287202Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6287314Z Ran 1 test in 1.656s
2022-11-23T02:14:29.6287333Z 
2022-11-23T02:14:29.6287440Z OK (skipped=1)
2022-11-23T02:14:29.6287458Z 
2022-11-23T02:14:29.6287584Z Generating XML reports...
2022-11-23T02:14:29.6288010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020106.xml
2022-11-23T02:14:29.6288380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6288553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6288985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6289179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6289198Z 
2022-11-23T02:14:29.6289308Z Running tests...
2022-11-23T02:14:29.6289569Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6289878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6290149Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6290370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30285
2022-11-23T02:14:29.6290583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30286
2022-11-23T02:14:29.6290947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6291121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6291493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6291678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6292045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6292213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6292565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6292840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6293085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6293334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6293730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6294120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6294346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6294582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6294800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6295022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6295419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6295806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6296061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclvgnw41
2022-11-23T02:14:29.6296312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs_z6l4k
2022-11-23T02:14:29.6296577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclvgnw41/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6296833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs_z6l4k/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6297068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6297300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6297560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6297791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6298153Z ok (7.759s)
2022-11-23T02:14:29.6298170Z 
2022-11-23T02:14:29.6298450Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6298563Z Ran 1 test in 7.759s
2022-11-23T02:14:29.6298583Z 
2022-11-23T02:14:29.6298674Z OK
2022-11-23T02:14:29.6298694Z 
2022-11-23T02:14:29.6298818Z Generating XML reports...
2022-11-23T02:14:29.6299266Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020111.xml
2022-11-23T02:14:29.6299616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6299796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6300175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6300365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6300384Z 
2022-11-23T02:14:29.6300491Z Running tests...
2022-11-23T02:14:29.6300751Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6301061Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6301330Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6301544Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30406
2022-11-23T02:14:29.6301740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30407
2022-11-23T02:14:29.6302192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6302365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6302742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6302929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6303289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6303458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6303829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6303996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6304240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6304477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6304878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6305269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6305496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6305771Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:14:29.6305992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6306258Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:14:29.6306497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponh53y_q
2022-11-23T02:14:29.6306826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponh53y_q/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6307086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8n7arw6p
2022-11-23T02:14:29.6307350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8n7arw6p/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6307582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6307812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6308036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6308268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6308547Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:14:29.6308803Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:14:29.6309070Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:14:29.6309334Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:14:29.6309562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6309795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6310023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6310250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6310571Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:14:29.6310842Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:14:29.6311096Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T02:14:29.6311361Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T02:14:29.6311589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6311816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6312042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6312268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6312373Z ok (8.063s)
2022-11-23T02:14:29.6312393Z 
2022-11-23T02:14:29.6312661Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6312757Z Ran 1 test in 8.063s
2022-11-23T02:14:29.6312796Z 
2022-11-23T02:14:29.6312871Z OK
2022-11-23T02:14:29.6312890Z 
2022-11-23T02:14:29.6313012Z Generating XML reports...
2022-11-23T02:14:29.6313461Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020121.xml
2022-11-23T02:14:29.6313826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6313998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6314373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6314567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6314586Z 
2022-11-23T02:14:29.6314695Z Running tests...
2022-11-23T02:14:29.6314938Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6315291Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6315566Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6316308Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2022-11-23T02:14:29.6316329Z 
2022-11-23T02:14:29.6316588Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6316700Z Ran 1 test in 1.630s
2022-11-23T02:14:29.6316723Z 
2022-11-23T02:14:29.6316834Z OK (skipped=1)
2022-11-23T02:14:29.6316853Z 
2022-11-23T02:14:29.6316975Z Generating XML reports...
2022-11-23T02:14:29.6317413Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020132.xml
2022-11-23T02:14:29.6317765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6317937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6318310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6318500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6318519Z 
2022-11-23T02:14:29.6318626Z Running tests...
2022-11-23T02:14:29.6318884Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6319194Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6319518Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6319737Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30559
2022-11-23T02:14:29.6319937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30560
2022-11-23T02:14:29.6320303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6320477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6320838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6321009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6321382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6321570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6321943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6322113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6322352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6322593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6322987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6323376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6323601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6324183Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:14:29.6324412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6324938Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:14:29.6325195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35711lhq
2022-11-23T02:14:29.6325462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35711lhq/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6325693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xv51xpa
2022-11-23T02:14:29.6326006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xv51xpa/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6326240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6326470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6326743Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T02:14:29.6327013Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T02:14:29.6327308Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T02:14:29.6327718Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T02:14:29.6328007Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T02:14:29.6328329Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T02:14:29.6328634Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T02:14:29.6328946Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T02:14:29.6329176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6329410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6329512Z ok (7.463s)
2022-11-23T02:14:29.6329532Z 
2022-11-23T02:14:29.6329804Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6329921Z Ran 1 test in 7.463s
2022-11-23T02:14:29.6329940Z 
2022-11-23T02:14:29.6330031Z OK
2022-11-23T02:14:29.6330050Z 
2022-11-23T02:14:29.6330173Z Generating XML reports...
2022-11-23T02:14:29.6331689Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020136.xml
2022-11-23T02:14:29.6332062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6332235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6332613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6332809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6332829Z 
2022-11-23T02:14:29.6332939Z Running tests...
2022-11-23T02:14:29.6333205Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6333575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6333968Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6333988Z 
2022-11-23T02:14:29.6334231Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6334346Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6334365Z 
2022-11-23T02:14:29.6334474Z OK (skipped=1)
2022-11-23T02:14:29.6334493Z 
2022-11-23T02:14:29.6334621Z Generating XML reports...
2022-11-23T02:14:29.6335066Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020146.xml
2022-11-23T02:14:29.6335436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6335613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6335991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6336183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6336202Z 
2022-11-23T02:14:29.6336294Z Running tests...
2022-11-23T02:14:29.6336559Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6336871Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6337258Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6337325Z 
2022-11-23T02:14:29.6337592Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6337706Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6337726Z 
2022-11-23T02:14:29.6337843Z OK (skipped=1)
2022-11-23T02:14:29.6337863Z 
2022-11-23T02:14:29.6338277Z Generating XML reports...
2022-11-23T02:14:29.6338732Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020148.xml
2022-11-23T02:14:29.6339081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6339261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6339635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6339824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6339848Z 
2022-11-23T02:14:29.6339958Z Running tests...
2022-11-23T02:14:29.6340221Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6340537Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6340981Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6341002Z 
2022-11-23T02:14:29.6341266Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6341360Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6341397Z 
2022-11-23T02:14:29.6341486Z OK (skipped=1)
2022-11-23T02:14:29.6341504Z 
2022-11-23T02:14:29.6341631Z Generating XML reports...
2022-11-23T02:14:29.6342079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020151.xml
2022-11-23T02:14:29.6342453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6342706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6343093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6343284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6343304Z 
2022-11-23T02:14:29.6343417Z Running tests...
2022-11-23T02:14:29.6343661Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6343973Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6344416Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6344441Z 
2022-11-23T02:14:29.6344701Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6344817Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6344836Z 
2022-11-23T02:14:29.6344944Z OK (skipped=1)
2022-11-23T02:14:29.6344962Z 
2022-11-23T02:14:29.6345090Z Generating XML reports...
2022-11-23T02:14:29.6345538Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020153.xml
2022-11-23T02:14:29.6345909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6346064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6346439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6346697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6346716Z 
2022-11-23T02:14:29.6346828Z Running tests...
2022-11-23T02:14:29.6347090Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6347408Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6347853Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6347873Z 
2022-11-23T02:14:29.6348135Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6348249Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6348267Z 
2022-11-23T02:14:29.6348376Z OK (skipped=1)
2022-11-23T02:14:29.6348395Z 
2022-11-23T02:14:29.6348500Z Generating XML reports...
2022-11-23T02:14:29.6348950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020155.xml
2022-11-23T02:14:29.6349317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6349495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6349872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6350063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6350082Z 
2022-11-23T02:14:29.6350191Z Running tests...
2022-11-23T02:14:29.6350454Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6350745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6351186Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6351211Z 
2022-11-23T02:14:29.6351534Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6351652Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6351671Z 
2022-11-23T02:14:29.6351781Z OK (skipped=1)
2022-11-23T02:14:29.6351800Z 
2022-11-23T02:14:29.6351926Z Generating XML reports...
2022-11-23T02:14:29.6352366Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020158.xml
2022-11-23T02:14:29.6352733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6352908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6353282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6353456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6353475Z 
2022-11-23T02:14:29.6353586Z Running tests...
2022-11-23T02:14:29.6353846Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6354158Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6354597Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6354617Z 
2022-11-23T02:14:29.6354880Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6354994Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6355013Z 
2022-11-23T02:14:29.6355123Z OK (skipped=1)
2022-11-23T02:14:29.6355142Z 
2022-11-23T02:14:29.6355318Z Generating XML reports...
2022-11-23T02:14:29.6355743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020200.xml
2022-11-23T02:14:29.6356112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6356288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6356665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6356857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6356876Z 
2022-11-23T02:14:29.6356991Z Running tests...
2022-11-23T02:14:29.6357253Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6357566Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6358008Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6358033Z 
2022-11-23T02:14:29.6358299Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6358394Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6358414Z 
2022-11-23T02:14:29.6358522Z OK (skipped=1)
2022-11-23T02:14:29.6358540Z 
2022-11-23T02:14:29.6358669Z Generating XML reports...
2022-11-23T02:14:29.6359113Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020202.xml
2022-11-23T02:14:29.6359479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6359655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6360028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6360221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6360240Z 
2022-11-23T02:14:29.6360330Z Running tests...
2022-11-23T02:14:29.6360635Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6360957Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6361395Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6361414Z 
2022-11-23T02:14:29.6361673Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6361788Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6361807Z 
2022-11-23T02:14:29.6361922Z OK (skipped=1)
2022-11-23T02:14:29.6361945Z 
2022-11-23T02:14:29.6362070Z Generating XML reports...
2022-11-23T02:14:29.6362514Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020205.xml
2022-11-23T02:14:29.6362867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6363043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6363420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6363611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6363630Z 
2022-11-23T02:14:29.6363740Z Running tests...
2022-11-23T02:14:29.6364002Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6364315Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6364803Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6364823Z 
2022-11-23T02:14:29.6365086Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6365199Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6365218Z 
2022-11-23T02:14:29.6365307Z OK (skipped=1)
2022-11-23T02:14:29.6365326Z 
2022-11-23T02:14:29.6365454Z Generating XML reports...
2022-11-23T02:14:29.6365896Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020207.xml
2022-11-23T02:14:29.6366263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6366437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6366820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6367009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6367029Z 
2022-11-23T02:14:29.6367142Z Running tests...
2022-11-23T02:14:29.6367383Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6367693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6368077Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6368097Z 
2022-11-23T02:14:29.6368357Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6368469Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6368488Z 
2022-11-23T02:14:29.6368596Z OK (skipped=1)
2022-11-23T02:14:29.6368620Z 
2022-11-23T02:14:29.6368746Z Generating XML reports...
2022-11-23T02:14:29.6369195Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020210.xml
2022-11-23T02:14:29.6369611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6369773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6370151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6370339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6370358Z 
2022-11-23T02:14:29.6370468Z Running tests...
2022-11-23T02:14:29.6370729Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6371039Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6371428Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:14:29.6371448Z 
2022-11-23T02:14:29.6371712Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6371827Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6371846Z 
2022-11-23T02:14:29.6371952Z OK (skipped=1)
2022-11-23T02:14:29.6371971Z 
2022-11-23T02:14:29.6372077Z Generating XML reports...
2022-11-23T02:14:29.6372522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020212.xml
2022-11-23T02:14:29.6372889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6373064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6373441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6373680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6373700Z 
2022-11-23T02:14:29.6373811Z Running tests...
2022-11-23T02:14:29.6374081Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6374372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6374635Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6375375Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.617s)
2022-11-23T02:14:29.6375395Z 
2022-11-23T02:14:29.6375660Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6375772Z Ran 1 test in 1.617s
2022-11-23T02:14:29.6375791Z 
2022-11-23T02:14:29.6375900Z OK (skipped=1)
2022-11-23T02:14:29.6375919Z 
2022-11-23T02:14:29.6376046Z Generating XML reports...
2022-11-23T02:14:29.6376495Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020214.xml
2022-11-23T02:14:29.6376864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6377037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6377394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6377583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6377602Z 
2022-11-23T02:14:29.6377712Z Running tests...
2022-11-23T02:14:29.6378145Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6378463Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6378786Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6379013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31108
2022-11-23T02:14:29.6379232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31109
2022-11-23T02:14:29.6379583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6379755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6380130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6380320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6380691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6380862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6381236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6381424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6381667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6381892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6382289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6382685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6382973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6383198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6383461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnl6cee5
2022-11-23T02:14:29.6383729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnl6cee5/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6383980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81x2txd6
2022-11-23T02:14:29.6384242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81x2txd6/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6384328Z ok (6.640s)
2022-11-23T02:14:29.6384348Z 
2022-11-23T02:14:29.6384614Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6384727Z Ran 1 test in 6.640s
2022-11-23T02:14:29.6384750Z 
2022-11-23T02:14:29.6384844Z OK
2022-11-23T02:14:29.6384863Z 
2022-11-23T02:14:29.6384989Z Generating XML reports...
2022-11-23T02:14:29.6385438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020219.xml
2022-11-23T02:14:29.6385807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6385984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6386340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6386529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6386548Z 
2022-11-23T02:14:29.6386657Z Running tests...
2022-11-23T02:14:29.6386921Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6387232Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6387507Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6387727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31222
2022-11-23T02:14:29.6387989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31223
2022-11-23T02:14:29.6388362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6388517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6388890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6389078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6389437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6389616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6389993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6390185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6390429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6390652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6391049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6391442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6391674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6391944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6392204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmper3yuddh
2022-11-23T02:14:29.6392481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmper3yuddh/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6392734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspgag_oj
2022-11-23T02:14:29.6393035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspgag_oj/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6393248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6393481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6393883Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmper3yuddh'>
2022-11-23T02:14:29.6394054Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:14:29.6394446Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpspgag_oj'>
2022-11-23T02:14:29.6394610Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:14:29.6394718Z ok (7.272s)
2022-11-23T02:14:29.6394738Z 
2022-11-23T02:14:29.6395004Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6395100Z Ran 1 test in 7.272s
2022-11-23T02:14:29.6395139Z 
2022-11-23T02:14:29.6395214Z OK
2022-11-23T02:14:29.6395233Z 
2022-11-23T02:14:29.6395356Z Generating XML reports...
2022-11-23T02:14:29.6395805Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020228.xml
2022-11-23T02:14:29.6396176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6396352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6396734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6396921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6396995Z 
2022-11-23T02:14:29.6397109Z Running tests...
2022-11-23T02:14:29.6397354Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6397663Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6397939Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-11-23T02:14:29.6397959Z 
2022-11-23T02:14:29.6398217Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6398332Z Ran 1 test in 0.006s
2022-11-23T02:14:29.6398351Z 
2022-11-23T02:14:29.6398460Z OK (skipped=1)
2022-11-23T02:14:29.6398482Z 
2022-11-23T02:14:29.6398605Z Generating XML reports...
2022-11-23T02:14:29.6399049Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020238.xml
2022-11-23T02:14:29.6399418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6399574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6399953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6400143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6400161Z 
2022-11-23T02:14:29.6400270Z Running tests...
2022-11-23T02:14:29.6400531Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6400843Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6401152Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6401371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31374
2022-11-23T02:14:29.6401570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31375
2022-11-23T02:14:29.6401940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6402115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6402488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6402677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6403037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6403208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6403580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6403765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6403992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6404235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6404633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6405024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6405252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6405470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6405729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8radgw4y
2022-11-23T02:14:29.6406002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8radgw4y/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6406281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptx4q_774
2022-11-23T02:14:29.6406551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptx4q_774/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6406784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6407017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6407123Z ok (7.326s)
2022-11-23T02:14:29.6407142Z 
2022-11-23T02:14:29.6407410Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6407525Z Ran 1 test in 7.326s
2022-11-23T02:14:29.6407548Z 
2022-11-23T02:14:29.6407641Z OK
2022-11-23T02:14:29.6407660Z 
2022-11-23T02:14:29.6407786Z Generating XML reports...
2022-11-23T02:14:29.6408212Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020240.xml
2022-11-23T02:14:29.6408582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6408759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6409134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6409322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6409341Z 
2022-11-23T02:14:29.6409451Z Running tests...
2022-11-23T02:14:29.6409710Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6410021Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6410336Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6410553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31493
2022-11-23T02:14:29.6410772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31494
2022-11-23T02:14:29.6411141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6411318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6411697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6411891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6412255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6412431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6412782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6412974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6413222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6413463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6413858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6414250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6414478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6414710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6414948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6415225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6415627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6416014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6416250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:14:29.6416482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:14:29.6416870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6417263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6417529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd98chih3
2022-11-23T02:14:29.6417795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd98chih3/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6418198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc4n5aed
2022-11-23T02:14:29.6418472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc4n5aed/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6418578Z ok (5.818s)
2022-11-23T02:14:29.6418599Z 
2022-11-23T02:14:29.6418869Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6418984Z Ran 1 test in 5.819s
2022-11-23T02:14:29.6419078Z 
2022-11-23T02:14:29.6419176Z OK
2022-11-23T02:14:29.6419195Z 
2022-11-23T02:14:29.6419325Z Generating XML reports...
2022-11-23T02:14:29.6419774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020250.xml
2022-11-23T02:14:29.6420149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6420305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6420680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6420870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6420890Z 
2022-11-23T02:14:29.6421000Z Running tests...
2022-11-23T02:14:29.6421261Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6421571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6421854Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6422070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31615
2022-11-23T02:14:29.6422269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31616
2022-11-23T02:14:29.6422641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6422815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6423190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6423384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6423745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6423924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6424294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6424538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6424771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6425010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6425410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6425802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6426033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6426263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6426503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6426746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6427136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6427506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6427739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:14:29.6427973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:14:29.6428366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6428807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6429068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn78mllkt
2022-11-23T02:14:29.6429324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht9k9iak
2022-11-23T02:14:29.6429594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn78mllkt/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6429857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht9k9iak/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6429943Z ok (16.681s)
2022-11-23T02:14:29.6429963Z 
2022-11-23T02:14:29.6430232Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6430347Z Ran 1 test in 16.682s
2022-11-23T02:14:29.6430367Z 
2022-11-23T02:14:29.6430464Z OK
2022-11-23T02:14:29.6430483Z 
2022-11-23T02:14:29.6430607Z Generating XML reports...
2022-11-23T02:14:29.6431054Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020258.xml
2022-11-23T02:14:29.6431428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6431605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6431963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6432154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6432173Z 
2022-11-23T02:14:29.6432283Z Running tests...
2022-11-23T02:14:29.6432544Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6432856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6433163Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6433380Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31737
2022-11-23T02:14:29.6433644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31738
2022-11-23T02:14:29.6434022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6434178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6434550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6434739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6435104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6435282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6435656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6435847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6436097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6436317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6436712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6437104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6437333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6437608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6437863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomlcz6of
2022-11-23T02:14:29.6438135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomlcz6of/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6438387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kicxl15
2022-11-23T02:14:29.6438651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kicxl15/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6438736Z ok (7.604s)
2022-11-23T02:14:29.6438755Z 
2022-11-23T02:14:29.6439023Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6439135Z Ran 1 test in 7.604s
2022-11-23T02:14:29.6439154Z 
2022-11-23T02:14:29.6439248Z OK
2022-11-23T02:14:29.6439266Z 
2022-11-23T02:14:29.6439391Z Generating XML reports...
2022-11-23T02:14:29.6439841Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020318.xml
2022-11-23T02:14:29.6440212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6440392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6440766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6440937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6440956Z 
2022-11-23T02:14:29.6441067Z Running tests...
2022-11-23T02:14:29.6441329Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6441641Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6441929Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6442150Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31856
2022-11-23T02:14:29.6442366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31857
2022-11-23T02:14:29.6442780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6442942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6443318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6443508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6443870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6444042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6444412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6444603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6444849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6445089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6445467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6445860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6446087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6446308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6446615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjro7grr4
2022-11-23T02:14:29.6446881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjro7grr4/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6447137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoehbp49
2022-11-23T02:14:29.6447402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoehbp49/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6447488Z ok (7.721s)
2022-11-23T02:14:29.6447507Z 
2022-11-23T02:14:29.6447774Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6447885Z Ran 1 test in 7.721s
2022-11-23T02:14:29.6447904Z 
2022-11-23T02:14:29.6447996Z OK
2022-11-23T02:14:29.6448015Z 
2022-11-23T02:14:29.6448137Z Generating XML reports...
2022-11-23T02:14:29.6448579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020328.xml
2022-11-23T02:14:29.6448950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6449121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6449482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6449669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6449688Z 
2022-11-23T02:14:29.6449794Z Running tests...
2022-11-23T02:14:29.6450051Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6450362Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6450616Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6450832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31975
2022-11-23T02:14:29.6451048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31976
2022-11-23T02:14:29.6451415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6451616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6451999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6452185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6452540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6452710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6453081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6453267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6453513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6453734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6454131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6454520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6454745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6454968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6455226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeu3she7u
2022-11-23T02:14:29.6455491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeu3she7u/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6455790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp931mowcn
2022-11-23T02:14:29.6456054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp931mowcn/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6456140Z ok (7.457s)
2022-11-23T02:14:29.6456160Z 
2022-11-23T02:14:29.6456424Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6456537Z Ran 1 test in 7.458s
2022-11-23T02:14:29.6456556Z 
2022-11-23T02:14:29.6456649Z OK
2022-11-23T02:14:29.6456668Z 
2022-11-23T02:14:29.6456791Z Generating XML reports...
2022-11-23T02:14:29.6457236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020338.xml
2022-11-23T02:14:29.6457600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6457778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6458321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6458497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6458516Z 
2022-11-23T02:14:29.6458626Z Running tests...
2022-11-23T02:14:29.6458889Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6459197Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6459456Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6459674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32090
2022-11-23T02:14:29.6459885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32091
2022-11-23T02:14:29.6460248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6460406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6460850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6461044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6461403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6461575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6461943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6462126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6462368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6462612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6462987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6463375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6463601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6463824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6464076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bi45zkw
2022-11-23T02:14:29.6464341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bi45zkw/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6464592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0gie0bb
2022-11-23T02:14:29.6464920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0gie0bb/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6465691Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:14:29.6466456Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:14:29.6466697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6466927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6467012Z ok (7.369s)
2022-11-23T02:14:29.6467032Z 
2022-11-23T02:14:29.6467298Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6467410Z Ran 1 test in 7.369s
2022-11-23T02:14:29.6467429Z 
2022-11-23T02:14:29.6467522Z OK
2022-11-23T02:14:29.6467540Z 
2022-11-23T02:14:29.6467663Z Generating XML reports...
2022-11-23T02:14:29.6468107Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020348.xml
2022-11-23T02:14:29.6468481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6468654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6469107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6469285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6469304Z 
2022-11-23T02:14:29.6469412Z Running tests...
2022-11-23T02:14:29.6469673Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6469982Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6470257Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6470995Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.647s)
2022-11-23T02:14:29.6471020Z 
2022-11-23T02:14:29.6471286Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6471397Z Ran 1 test in 1.647s
2022-11-23T02:14:29.6471417Z 
2022-11-23T02:14:29.6471524Z OK (skipped=1)
2022-11-23T02:14:29.6471543Z 
2022-11-23T02:14:29.6471649Z Generating XML reports...
2022-11-23T02:14:29.6472090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020358.xml
2022-11-23T02:14:29.6472452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6472625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6473053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6473241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6473260Z 
2022-11-23T02:14:29.6473369Z Running tests...
2022-11-23T02:14:29.6473632Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6473943Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6474204Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6474942Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.654s)
2022-11-23T02:14:29.6474966Z 
2022-11-23T02:14:29.6475226Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6475337Z Ran 1 test in 1.654s
2022-11-23T02:14:29.6475357Z 
2022-11-23T02:14:29.6475462Z OK (skipped=1)
2022-11-23T02:14:29.6475480Z 
2022-11-23T02:14:29.6475603Z Generating XML reports...
2022-11-23T02:14:29.6476047Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020402.xml
2022-11-23T02:14:29.6476411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6476585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6476958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6477128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6477148Z 
2022-11-23T02:14:29.6477254Z Running tests...
2022-11-23T02:14:29.6477518Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6477826Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6478156Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6478378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32277
2022-11-23T02:14:29.6478593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32278
2022-11-23T02:14:29.6478955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6479111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6479483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6479668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6480034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6480205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6480573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6480758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6480998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6481236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6481615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6482006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6482283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6482508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6482763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposy5wyr9
2022-11-23T02:14:29.6483031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposy5wyr9/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6483280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9frjkn5
2022-11-23T02:14:29.6483543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9frjkn5/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6483863Z STAGE:2022-11-23 02:04:12 32278:32278 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6484180Z STAGE:2022-11-23 02:04:12 32277:32277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6484419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6484651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6484985Z STAGE:2022-11-23 02:04:12 32278:32278 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6485306Z STAGE:2022-11-23 02:04:12 32277:32277 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6485647Z STAGE:2022-11-23 02:04:12 32277:32277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6485989Z STAGE:2022-11-23 02:04:12 32278:32278 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6486806Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:14:29.6487583Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:14:29.6487917Z STAGE:2022-11-23 02:04:12 32277:32277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6488239Z STAGE:2022-11-23 02:04:13 32278:32278 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6488569Z STAGE:2022-11-23 02:04:13 32277:32277 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6488896Z STAGE:2022-11-23 02:04:13 32277:32277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6489224Z STAGE:2022-11-23 02:04:13 32278:32278 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6489567Z STAGE:2022-11-23 02:04:13 32278:32278 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6489671Z ok (6.749s)
2022-11-23T02:14:29.6489690Z 
2022-11-23T02:14:29.6489951Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6490063Z Ran 1 test in 6.749s
2022-11-23T02:14:29.6490082Z 
2022-11-23T02:14:29.6490174Z OK
2022-11-23T02:14:29.6490192Z 
2022-11-23T02:14:29.6490365Z Generating XML reports...
2022-11-23T02:14:29.6490813Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020406.xml
2022-11-23T02:14:29.6491165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6491338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6491715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6491903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6491922Z 
2022-11-23T02:14:29.6492030Z Running tests...
2022-11-23T02:14:29.6492288Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6492599Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6492897Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6493104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32399
2022-11-23T02:14:29.6493318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32400
2022-11-23T02:14:29.6493688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6493859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6494231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6494416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6494774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6494944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6495324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6495493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6495778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6496026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6496422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6496812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6497040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6497265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6497523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxh45r_xr
2022-11-23T02:14:29.6497769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxh45r_xr/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6498237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpew3hxwm1
2022-11-23T02:14:29.6498513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpew3hxwm1/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6498615Z ok (6.866s)
2022-11-23T02:14:29.6498634Z 
2022-11-23T02:14:29.6498904Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6499016Z Ran 1 test in 6.867s
2022-11-23T02:14:29.6499035Z 
2022-11-23T02:14:29.6499127Z OK
2022-11-23T02:14:29.6499146Z 
2022-11-23T02:14:29.6499270Z Generating XML reports...
2022-11-23T02:14:29.6499715Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020416.xml
2022-11-23T02:14:29.6500146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6500319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6500699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6500886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6500905Z 
2022-11-23T02:14:29.6501014Z Running tests...
2022-11-23T02:14:29.6501274Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6501583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6501853Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6502593Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.607s)
2022-11-23T02:14:29.6502618Z 
2022-11-23T02:14:29.6502880Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6502975Z Ran 1 test in 1.607s
2022-11-23T02:14:29.6502994Z 
2022-11-23T02:14:29.6503102Z OK (skipped=1)
2022-11-23T02:14:29.6503121Z 
2022-11-23T02:14:29.6503245Z Generating XML reports...
2022-11-23T02:14:29.6503687Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020425.xml
2022-11-23T02:14:29.6504055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6504230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6504610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6504800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6504819Z 
2022-11-23T02:14:29.6504909Z Running tests...
2022-11-23T02:14:29.6505230Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6505552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6505828Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6506042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32548
2022-11-23T02:14:29.6506256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32549
2022-11-23T02:14:29.6506619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6506795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6507168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6507341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6507697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6507867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6508233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6508418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6508661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6508902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6509349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6509724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6509950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6510172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6510425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwdi_c7m
2022-11-23T02:14:29.6510689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwdi_c7m/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6510939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaurpnkjg
2022-11-23T02:14:29.6511208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaurpnkjg/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6512117Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:14:29.6512231Z   warnings.warn(
2022-11-23T02:14:29.6513131Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:14:29.6513246Z   warnings.warn(
2022-11-23T02:14:29.6513464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6513687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6513787Z ok (6.364s)
2022-11-23T02:14:29.6513806Z 
2022-11-23T02:14:29.6514115Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6514232Z Ran 1 test in 6.364s
2022-11-23T02:14:29.6514251Z 
2022-11-23T02:14:29.6514344Z OK
2022-11-23T02:14:29.6514362Z 
2022-11-23T02:14:29.6514484Z Generating XML reports...
2022-11-23T02:14:29.6514929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020429.xml
2022-11-23T02:14:29.6515279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6515452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6515828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6516016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6516035Z 
2022-11-23T02:14:29.6516142Z Running tests...
2022-11-23T02:14:29.6516407Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6516720Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6516992Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6517724Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.652s)
2022-11-23T02:14:29.6517791Z 
2022-11-23T02:14:29.6518053Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6518148Z Ran 1 test in 1.652s
2022-11-23T02:14:29.6518167Z 
2022-11-23T02:14:29.6518273Z OK (skipped=1)
2022-11-23T02:14:29.6518292Z 
2022-11-23T02:14:29.6518417Z Generating XML reports...
2022-11-23T02:14:29.6518862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020438.xml
2022-11-23T02:14:29.6519229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6519401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6519779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6519967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6519987Z 
2022-11-23T02:14:29.6520096Z Running tests...
2022-11-23T02:14:29.6520341Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6520649Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6520920Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6521133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32700
2022-11-23T02:14:29.6521347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32701
2022-11-23T02:14:29.6521710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6521885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6522258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6522433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6522792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6522961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6523372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6523563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6523806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6524047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6524445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6524834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6525048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6525273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6525524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcscnikn
2022-11-23T02:14:29.6525790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcscnikn/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6526038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpieltxaw5
2022-11-23T02:14:29.6526298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpieltxaw5/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6526632Z STAGE:2022-11-23 02:04:48 32700:32700 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6526960Z STAGE:2022-11-23 02:04:48 32701:32701 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6527636Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.6527733Z   warnings.warn(
2022-11-23T02:14:29.6528346Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:14:29.6528455Z   warnings.warn(
2022-11-23T02:14:29.6528688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6528918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:14:29.6529255Z STAGE:2022-11-23 02:04:48 32700:32700 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6529582Z STAGE:2022-11-23 02:04:48 32701:32701 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6529926Z STAGE:2022-11-23 02:04:48 32700:32700 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6530271Z STAGE:2022-11-23 02:04:48 32701:32701 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6530581Z STAGE:2022-11-23 02:04:48 32700:32700 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6530913Z STAGE:2022-11-23 02:04:49 32700:32700 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6531255Z STAGE:2022-11-23 02:04:49 32700:32700 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6531358Z ok (7.126s)
2022-11-23T02:14:29.6531378Z 
2022-11-23T02:14:29.6531639Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6531753Z Ran 1 test in 7.126s
2022-11-23T02:14:29.6531772Z 
2022-11-23T02:14:29.6531863Z OK
2022-11-23T02:14:29.6531882Z 
2022-11-23T02:14:29.6532005Z Generating XML reports...
2022-11-23T02:14:29.6532482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020442.xml
2022-11-23T02:14:29.6532857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6533031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6533404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6533590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6533609Z 
2022-11-23T02:14:29.6533716Z Running tests...
2022-11-23T02:14:29.6533973Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6534287Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6534549Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6534753Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32822
2022-11-23T02:14:29.6534964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32823
2022-11-23T02:14:29.6535327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6535498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6535869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6536057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6536416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6536634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6536986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6537175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6537412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6537651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6538277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6538673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6538898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6539127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6539380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t79ycbx
2022-11-23T02:14:29.6539632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t79ycbx/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6539884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikdcjtk_
2022-11-23T02:14:29.6540144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikdcjtk_/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6540245Z ok (7.057s)
2022-11-23T02:14:29.6540265Z 
2022-11-23T02:14:29.6540529Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6540641Z Ran 1 test in 7.057s
2022-11-23T02:14:29.6540660Z 
2022-11-23T02:14:29.6540750Z OK
2022-11-23T02:14:29.6540769Z 
2022-11-23T02:14:29.6540896Z Generating XML reports...
2022-11-23T02:14:29.6541338Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020452.xml
2022-11-23T02:14:29.6541768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6541952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6542332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6542519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6542538Z 
2022-11-23T02:14:29.6542647Z Running tests...
2022-11-23T02:14:29.6542906Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6543215Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6543484Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6543687Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32937
2022-11-23T02:14:29.6543898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32938
2022-11-23T02:14:29.6544262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6544433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6544805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6544992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6545347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6545518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6545955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6546122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6546368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6546608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6546999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6547390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6547616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6547841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6548098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp757lw98r
2022-11-23T02:14:29.6548342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp757lw98r/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6548599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0155ssl
2022-11-23T02:14:29.6548860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0155ssl/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6548962Z ok (6.976s)
2022-11-23T02:14:29.6548982Z 
2022-11-23T02:14:29.6549247Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6549358Z Ran 1 test in 6.976s
2022-11-23T02:14:29.6549377Z 
2022-11-23T02:14:29.6549470Z OK
2022-11-23T02:14:29.6549488Z 
2022-11-23T02:14:29.6549611Z Generating XML reports...
2022-11-23T02:14:29.6550051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020502.xml
2022-11-23T02:14:29.6550402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6550575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6550994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6551185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6551205Z 
2022-11-23T02:14:29.6551314Z Running tests...
2022-11-23T02:14:29.6551571Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6551881Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6552149Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6552886Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-11-23T02:14:29.6552911Z 
2022-11-23T02:14:29.6553170Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6553264Z Ran 1 test in 1.644s
2022-11-23T02:14:29.6553283Z 
2022-11-23T02:14:29.6553390Z OK (skipped=1)
2022-11-23T02:14:29.6553409Z 
2022-11-23T02:14:29.6553533Z Generating XML reports...
2022-11-23T02:14:29.6553978Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020511.xml
2022-11-23T02:14:29.6554342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6554516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6554943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6555129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6555148Z 
2022-11-23T02:14:29.6555243Z Running tests...
2022-11-23T02:14:29.6555504Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6555814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6556067Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6556807Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.660s)
2022-11-23T02:14:29.6556831Z 
2022-11-23T02:14:29.6557091Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6557202Z Ran 1 test in 1.660s
2022-11-23T02:14:29.6557221Z 
2022-11-23T02:14:29.6557328Z OK (skipped=1)
2022-11-23T02:14:29.6557346Z 
2022-11-23T02:14:29.6557472Z Generating XML reports...
2022-11-23T02:14:29.6557914Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020515.xml
2022-11-23T02:14:29.6558262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6558436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6558808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6558994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6559013Z 
2022-11-23T02:14:29.6559123Z Running tests...
2022-11-23T02:14:29.6559384Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6559695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6560027Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6560771Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.610s)
2022-11-23T02:14:29.6560792Z 
2022-11-23T02:14:29.6561048Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6561142Z Ran 1 test in 1.610s
2022-11-23T02:14:29.6561161Z 
2022-11-23T02:14:29.6561267Z OK (skipped=1)
2022-11-23T02:14:29.6561286Z 
2022-11-23T02:14:29.6561411Z Generating XML reports...
2022-11-23T02:14:29.6561851Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020519.xml
2022-11-23T02:14:29.6562218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6562392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6562765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6562953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6562972Z 
2022-11-23T02:14:29.6563062Z Running tests...
2022-11-23T02:14:29.6563321Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6563628Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6563972Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6564188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33154
2022-11-23T02:14:29.6564407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33155
2022-11-23T02:14:29.6564775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6564948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6565321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6565491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6565849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6566020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6566393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6566580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6566825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6567065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6567461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6567830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6568059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6568283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6568537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zbqnb61
2022-11-23T02:14:29.6568847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zbqnb61/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6569103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmnp5zm7
2022-11-23T02:14:29.6569364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmnp5zm7/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6569467Z ok (7.370s)
2022-11-23T02:14:29.6569487Z 
2022-11-23T02:14:29.6569750Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6569844Z Ran 1 test in 7.370s
2022-11-23T02:14:29.6569864Z 
2022-11-23T02:14:29.6569954Z OK
2022-11-23T02:14:29.6569973Z 
2022-11-23T02:14:29.6570096Z Generating XML reports...
2022-11-23T02:14:29.6570544Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020523.xml
2022-11-23T02:14:29.6570911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6571084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6571462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6571651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6571670Z 
2022-11-23T02:14:29.6571778Z Running tests...
2022-11-23T02:14:29.6572020Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6572333Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6572599Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6572869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33273
2022-11-23T02:14:29.6573085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33274
2022-11-23T02:14:29.6573451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6573625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6573996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6574164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6574517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6574685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6575057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6575245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6575488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6575731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6576128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6576517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6576727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6577101Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T02:14:29.6577354Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T02:14:29.6577584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6578116Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T02:14:29.6578443Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T02:14:29.6578701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0a7gxra
2022-11-23T02:14:29.6578969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0a7gxra/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6579204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpem_xhe
2022-11-23T02:14:29.6579466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpem_xhe/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6579568Z ok (6.965s)
2022-11-23T02:14:29.6579588Z 
2022-11-23T02:14:29.6579854Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6579970Z Ran 1 test in 6.965s
2022-11-23T02:14:29.6579989Z 
2022-11-23T02:14:29.6580082Z OK
2022-11-23T02:14:29.6580100Z 
2022-11-23T02:14:29.6580224Z Generating XML reports...
2022-11-23T02:14:29.6580671Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020533.xml
2022-11-23T02:14:29.6581037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6581193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6581564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6581752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6581771Z 
2022-11-23T02:14:29.6581878Z Running tests...
2022-11-23T02:14:29.6582137Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6582514Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6582771Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6582992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33388
2022-11-23T02:14:29.6583189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33389
2022-11-23T02:14:29.6583554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6583730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6584100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6584285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6584652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6584823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6585198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6585381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6585604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6585845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6586241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6586630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6586858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6587097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6587361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6587600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6587992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6588363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6588466Z ok (4.311s)
2022-11-23T02:14:29.6588485Z 
2022-11-23T02:14:29.6588747Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6588861Z Ran 1 test in 4.311s
2022-11-23T02:14:29.6588880Z 
2022-11-23T02:14:29.6588972Z OK
2022-11-23T02:14:29.6588990Z 
2022-11-23T02:14:29.6589112Z Generating XML reports...
2022-11-23T02:14:29.6589558Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020543.xml
2022-11-23T02:14:29.6589926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6590081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6590455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6590643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6590662Z 
2022-11-23T02:14:29.6590769Z Running tests...
2022-11-23T02:14:29.6591027Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6591394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6591641Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6591856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33495
2022-11-23T02:14:29.6592070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33496
2022-11-23T02:14:29.6592420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6592592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6593016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6593202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6593562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6593735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6594106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6594294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6594519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6594755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6595150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6595539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6595768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6596007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6596227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6596502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6596901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6597270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6597373Z ok (4.414s)
2022-11-23T02:14:29.6597393Z 
2022-11-23T02:14:29.6597651Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6597760Z Ran 1 test in 4.415s
2022-11-23T02:14:29.6597780Z 
2022-11-23T02:14:29.6597871Z OK
2022-11-23T02:14:29.6597893Z 
2022-11-23T02:14:29.6598017Z Generating XML reports...
2022-11-23T02:14:29.6598463Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020550.xml
2022-11-23T02:14:29.6598832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6599007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6599366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6599553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6599572Z 
2022-11-23T02:14:29.6599680Z Running tests...
2022-11-23T02:14:29.6599939Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6600248Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6600567Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6601308Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2022-11-23T02:14:29.6601329Z 
2022-11-23T02:14:29.6601586Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6601698Z Ran 1 test in 1.621s
2022-11-23T02:14:29.6601716Z 
2022-11-23T02:14:29.6601823Z OK (skipped=1)
2022-11-23T02:14:29.6601842Z 
2022-11-23T02:14:29.6601948Z Generating XML reports...
2022-11-23T02:14:29.6602388Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020557.xml
2022-11-23T02:14:29.6602755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6602931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6603311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6603498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6603518Z 
2022-11-23T02:14:29.6603626Z Running tests...
2022-11-23T02:14:29.6603885Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6604180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6604450Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6605184Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2022-11-23T02:14:29.6605208Z 
2022-11-23T02:14:29.6605469Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6605635Z Ran 1 test in 1.622s
2022-11-23T02:14:29.6605656Z 
2022-11-23T02:14:29.6605765Z OK (skipped=1)
2022-11-23T02:14:29.6605784Z 
2022-11-23T02:14:29.6605907Z Generating XML reports...
2022-11-23T02:14:29.6606353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020601.xml
2022-11-23T02:14:29.6606719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6606891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6607248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6607439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6607458Z 
2022-11-23T02:14:29.6607564Z Running tests...
2022-11-23T02:14:29.6607822Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6608134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6608400Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6608618Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33670
2022-11-23T02:14:29.6608834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33671
2022-11-23T02:14:29.6609185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6609360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6609818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6610002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6610361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6610531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6610900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6611090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6611329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6611551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6611954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6612345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6612576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6612800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6612900Z ok (4.323s)
2022-11-23T02:14:29.6612920Z 
2022-11-23T02:14:29.6613181Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6613292Z Ran 1 test in 4.324s
2022-11-23T02:14:29.6613311Z 
2022-11-23T02:14:29.6613386Z OK
2022-11-23T02:14:29.6613422Z 
2022-11-23T02:14:29.6613527Z Generating XML reports...
2022-11-23T02:14:29.6613969Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020605.xml
2022-11-23T02:14:29.6614338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6614511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6614934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6615125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6615144Z 
2022-11-23T02:14:29.6615252Z Running tests...
2022-11-23T02:14:29.6615513Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6615806Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6616054Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.6616073Z 
2022-11-23T02:14:29.6616328Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6616443Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6616462Z 
2022-11-23T02:14:29.6616568Z OK (skipped=1)
2022-11-23T02:14:29.6616587Z 
2022-11-23T02:14:29.6616709Z Generating XML reports...
2022-11-23T02:14:29.6617156Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020612.xml
2022-11-23T02:14:29.6617518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6617692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6618258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6618452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6618472Z 
2022-11-23T02:14:29.6618581Z Running tests...
2022-11-23T02:14:29.6618844Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6619232Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6619487Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.6619510Z 
2022-11-23T02:14:29.6619766Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6619876Z Ran 1 test in 0.003s
2022-11-23T02:14:29.6619895Z 
2022-11-23T02:14:29.6620003Z OK (skipped=1)
2022-11-23T02:14:29.6620023Z 
2022-11-23T02:14:29.6620127Z Generating XML reports...
2022-11-23T02:14:29.6620569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020614.xml
2022-11-23T02:14:29.6620934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6621105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6621483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6621671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6621690Z 
2022-11-23T02:14:29.6621801Z Running tests...
2022-11-23T02:14:29.6622066Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6622360Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6622605Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6622820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33841
2022-11-23T02:14:29.6623031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33842
2022-11-23T02:14:29.6623398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6623573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6623946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6624193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6624562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6624717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6625082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6625264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6625506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6625788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6626188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6626580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6626805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6627013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6627346Z STAGE:2022-11-23 02:06:22 33842:33842 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6627667Z STAGE:2022-11-23 02:06:22 33841:33841 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6627995Z STAGE:2022-11-23 02:06:24 33841:33841 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6628394Z STAGE:2022-11-23 02:06:24 33841:33841 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6628724Z STAGE:2022-11-23 02:06:24 33842:33842 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6629068Z STAGE:2022-11-23 02:06:24 33842:33842 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6629391Z STAGE:2022-11-23 02:06:24 33841:33841 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6629713Z STAGE:2022-11-23 02:06:24 33842:33842 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.6630022Z STAGE:2022-11-23 02:06:24 33841:33841 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6630360Z STAGE:2022-11-23 02:06:24 33841:33841 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6630685Z STAGE:2022-11-23 02:06:24 33842:33842 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.6631027Z STAGE:2022-11-23 02:06:24 33842:33842 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.6631128Z ok (7.849s)
2022-11-23T02:14:29.6631147Z 
2022-11-23T02:14:29.6631411Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6631521Z Ran 1 test in 7.849s
2022-11-23T02:14:29.6631541Z 
2022-11-23T02:14:29.6631632Z OK
2022-11-23T02:14:29.6631650Z 
2022-11-23T02:14:29.6631772Z Generating XML reports...
2022-11-23T02:14:29.6632199Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020617.xml
2022-11-23T02:14:29.6632562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6632735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6633106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6633297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6633316Z 
2022-11-23T02:14:29.6633423Z Running tests...
2022-11-23T02:14:29.6633726Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6634053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6634297Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.6634333Z 
2022-11-23T02:14:29.6634569Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6634679Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6634698Z 
2022-11-23T02:14:29.6634805Z OK (skipped=1)
2022-11-23T02:14:29.6634824Z 
2022-11-23T02:14:29.6634944Z Generating XML reports...
2022-11-23T02:14:29.6635386Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020627.xml
2022-11-23T02:14:29.6635752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6635925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6636300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6636471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6636506Z 
2022-11-23T02:14:29.6636595Z Running tests...
2022-11-23T02:14:29.6636853Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6637162Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6637417Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.6637483Z 
2022-11-23T02:14:29.6637749Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6637860Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6637880Z 
2022-11-23T02:14:29.6637988Z OK (skipped=1)
2022-11-23T02:14:29.6638006Z 
2022-11-23T02:14:29.6638128Z Generating XML reports...
2022-11-23T02:14:29.6638556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020629.xml
2022-11-23T02:14:29.6638920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6639095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6639469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6639654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6639673Z 
2022-11-23T02:14:29.6639784Z Running tests...
2022-11-23T02:14:29.6640044Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6640352Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6640601Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6640800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34030
2022-11-23T02:14:29.6641016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34031
2022-11-23T02:14:29.6641381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6641556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6641927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6642116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6642486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6642936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6643550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6644021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6644470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6644942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6645585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6646269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6646778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6647226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6647571Z ok (7.065s)
2022-11-23T02:14:29.6647717Z 
2022-11-23T02:14:29.6647983Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6648287Z Ran 1 test in 7.065s
2022-11-23T02:14:29.6648447Z 
2022-11-23T02:14:29.6648544Z OK
2022-11-23T02:14:29.6648678Z 
2022-11-23T02:14:29.6648801Z Generating XML reports...
2022-11-23T02:14:29.6649401Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020632.xml
2022-11-23T02:14:29.6650093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6650539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6651157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6651600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6651829Z 
2022-11-23T02:14:29.6651937Z Running tests...
2022-11-23T02:14:29.6652333Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6652861Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6653370Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6654391Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.608s)
2022-11-23T02:14:29.6654917Z 
2022-11-23T02:14:29.6655181Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6655505Z Ran 1 test in 1.608s
2022-11-23T02:14:29.6655664Z 
2022-11-23T02:14:29.6655755Z OK (skipped=1)
2022-11-23T02:14:29.6655911Z 
2022-11-23T02:14:29.6656038Z Generating XML reports...
2022-11-23T02:14:29.6656649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020641.xml
2022-11-23T02:14:29.6657379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6657806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6658551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6659013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6659244Z 
2022-11-23T02:14:29.6659352Z Running tests...
2022-11-23T02:14:29.6659729Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6660251Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6660816Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6661283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34182
2022-11-23T02:14:29.6661724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34183
2022-11-23T02:14:29.6662316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6662754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6663299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6663759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6664324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6664764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6665305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6665761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6666202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6666689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6667339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6668092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6668610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6669089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6669570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6670041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6670691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6671349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6671738Z ok (4.269s)
2022-11-23T02:14:29.6671883Z 
2022-11-23T02:14:29.6672153Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6672475Z Ran 1 test in 4.269s
2022-11-23T02:14:29.6672615Z 
2022-11-23T02:14:29.6672710Z OK
2022-11-23T02:14:29.6672841Z 
2022-11-23T02:14:29.6672964Z Generating XML reports...
2022-11-23T02:14:29.6673562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020646.xml
2022-11-23T02:14:29.6674247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6674690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6675255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6675718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6675938Z 
2022-11-23T02:14:29.6676031Z Running tests...
2022-11-23T02:14:29.6676425Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6676950Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6677439Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6677980Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34289
2022-11-23T02:14:29.6678452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34290
2022-11-23T02:14:29.6679058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6679507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6680059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6680523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6681097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6681533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6682083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6682540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6682989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6683474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6684132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6684811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6685408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6685891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6686217Z ok (6.959s)
2022-11-23T02:14:29.6686368Z 
2022-11-23T02:14:29.6686638Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6686961Z Ran 1 test in 6.959s
2022-11-23T02:14:29.6687102Z 
2022-11-23T02:14:29.6687206Z OK
2022-11-23T02:14:29.6687339Z 
2022-11-23T02:14:29.6687461Z Generating XML reports...
2022-11-23T02:14:29.6688063Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020652.xml
2022-11-23T02:14:29.6688768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6689218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6689789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6690256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6690496Z 
2022-11-23T02:14:29.6690607Z Running tests...
2022-11-23T02:14:29.6691000Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6691528Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6692020Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6692472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34404
2022-11-23T02:14:29.6692956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34405
2022-11-23T02:14:29.6693564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6694018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6694566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6695092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6695674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6696092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6696773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6697510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6698148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6698792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6699377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6700067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6700588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6701035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6701375Z ok (4.198s)
2022-11-23T02:14:29.6701523Z 
2022-11-23T02:14:29.6701788Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6702094Z Ran 1 test in 4.198s
2022-11-23T02:14:29.6702253Z 
2022-11-23T02:14:29.6702348Z OK
2022-11-23T02:14:29.6702481Z 
2022-11-23T02:14:29.6702605Z Generating XML reports...
2022-11-23T02:14:29.6703276Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020702.xml
2022-11-23T02:14:29.6703983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6704433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6705002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6705448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6705676Z 
2022-11-23T02:14:29.6705786Z Running tests...
2022-11-23T02:14:29.6706184Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6706714Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6707210Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6707701Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34509
2022-11-23T02:14:29.6708149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34510
2022-11-23T02:14:29.6708734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6709181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6709749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6710216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6710767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6711207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6711765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6712207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6712712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6713216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6713888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6714555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6715070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6715547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6716029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6716485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6717133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6717807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6718199Z ok (4.360s)
2022-11-23T02:14:29.6718329Z 
2022-11-23T02:14:29.6718594Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6718916Z Ran 1 test in 4.360s
2022-11-23T02:14:29.6719075Z 
2022-11-23T02:14:29.6719167Z OK
2022-11-23T02:14:29.6719282Z 
2022-11-23T02:14:29.6719408Z Generating XML reports...
2022-11-23T02:14:29.6720007Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020709.xml
2022-11-23T02:14:29.6720774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6721218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6721766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6722232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6722460Z 
2022-11-23T02:14:29.6722570Z Running tests...
2022-11-23T02:14:29.6722954Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6723474Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6724063Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6724545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34616
2022-11-23T02:14:29.6724980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34617
2022-11-23T02:14:29.6725574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6726020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6726570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6727032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6727604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6728044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6728588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6729051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6729499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6730018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6730676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6731411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6732138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6732601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6733074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6733551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6734200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6734861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6735248Z ok (4.458s)
2022-11-23T02:14:29.6735396Z 
2022-11-23T02:14:29.6735661Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6735969Z Ran 1 test in 4.458s
2022-11-23T02:14:29.6736130Z 
2022-11-23T02:14:29.6736224Z OK
2022-11-23T02:14:29.6736356Z 
2022-11-23T02:14:29.6736480Z Generating XML reports...
2022-11-23T02:14:29.6737083Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020716.xml
2022-11-23T02:14:29.6737771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6738477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6739051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6739513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6739720Z 
2022-11-23T02:14:29.6739829Z Running tests...
2022-11-23T02:14:29.6740227Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6740750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6741237Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6741891Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34723
2022-11-23T02:14:29.6742343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34724
2022-11-23T02:14:29.6742954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6743384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6744014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6744552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6745110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6745555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6746115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6746572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6747005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6747496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6748226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6748900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6749415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6749878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6750370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b3qgnw2
2022-11-23T02:14:29.6750883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b3qgnw2/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6751409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxdqncnq
2022-11-23T02:14:29.6751941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxdqncnq/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6752319Z ok (6.368s)
2022-11-23T02:14:29.6752449Z 
2022-11-23T02:14:29.6752721Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6753046Z Ran 1 test in 6.368s
2022-11-23T02:14:29.6753206Z 
2022-11-23T02:14:29.6753300Z OK
2022-11-23T02:14:29.6753432Z 
2022-11-23T02:14:29.6753537Z Generating XML reports...
2022-11-23T02:14:29.6754142Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020723.xml
2022-11-23T02:14:29.6754844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6755284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6755907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6756365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6756593Z 
2022-11-23T02:14:29.6756703Z Running tests...
2022-11-23T02:14:29.6757084Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6757608Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6758085Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-11-23T02:14:29.6758351Z 
2022-11-23T02:14:29.6758610Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6758910Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6759068Z 
2022-11-23T02:14:29.6759177Z OK (skipped=1)
2022-11-23T02:14:29.6759330Z 
2022-11-23T02:14:29.6759460Z Generating XML reports...
2022-11-23T02:14:29.6760044Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020731.xml
2022-11-23T02:14:29.6760758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6761201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6761773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6762219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6762444Z 
2022-11-23T02:14:29.6762554Z Running tests...
2022-11-23T02:14:29.6762950Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6763455Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6763932Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T02:14:29.6764201Z 
2022-11-23T02:14:29.6764462Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6764781Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6764939Z 
2022-11-23T02:14:29.6765079Z OK (skipped=1)
2022-11-23T02:14:29.6765240Z 
2022-11-23T02:14:29.6765364Z Generating XML reports...
2022-11-23T02:14:29.6765961Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020734.xml
2022-11-23T02:14:29.6766839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6767335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6768000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6768588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6768865Z 
2022-11-23T02:14:29.6769049Z Running tests...
2022-11-23T02:14:29.6769531Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6770079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6770671Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T02:14:29.6771041Z 
2022-11-23T02:14:29.6771341Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6771736Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6771880Z 
2022-11-23T02:14:29.6772024Z OK (skipped=1)
2022-11-23T02:14:29.6772211Z 
2022-11-23T02:14:29.6772383Z Generating XML reports...
2022-11-23T02:14:29.6773092Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020736.xml
2022-11-23T02:14:29.6773857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6774452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6775115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6775651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6775954Z 
2022-11-23T02:14:29.6776050Z Running tests...
2022-11-23T02:14:29.6776525Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6777133Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6777656Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T02:14:29.6778174Z 
2022-11-23T02:14:29.6778480Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6778965Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6779172Z 
2022-11-23T02:14:29.6779318Z OK (skipped=1)
2022-11-23T02:14:29.6779509Z 
2022-11-23T02:14:29.6779617Z Generating XML reports...
2022-11-23T02:14:29.6780298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020739.xml
2022-11-23T02:14:29.6781113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6781639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6782229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6782762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6783026Z 
2022-11-23T02:14:29.6783170Z Running tests...
2022-11-23T02:14:29.6783668Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6784268Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6784880Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6785530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34973
2022-11-23T02:14:29.6786017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34974
2022-11-23T02:14:29.6786735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6787251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6787840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6788388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6789099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6789612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6790202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6790754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6791273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6791820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6792547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6793366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6793994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6794573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6795168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6795731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6796428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6810677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6811222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:14:29.6811715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:14:29.6812352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6813029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6813553Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:14:29.6813899Z ok (22.285s)
2022-11-23T02:14:29.6814034Z 
2022-11-23T02:14:29.6814301Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6814628Z Ran 1 test in 22.285s
2022-11-23T02:14:29.6814787Z 
2022-11-23T02:14:29.6814882Z OK
2022-11-23T02:14:29.6815012Z 
2022-11-23T02:14:29.6815119Z Generating XML reports...
2022-11-23T02:14:29.6815724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020741.xml
2022-11-23T02:14:29.6816437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6816889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6817440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6818305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6818558Z 
2022-11-23T02:14:29.6818669Z Running tests...
2022-11-23T02:14:29.6819059Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6819581Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6820123Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6820640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35096
2022-11-23T02:14:29.6821069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35097
2022-11-23T02:14:29.6821666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6822112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6822669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6823138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6823709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6824145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6824688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6825144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6825671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6826146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6826801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6827484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6827998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6828501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.6829186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6829661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.6830321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6830981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.6831510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:14:29.6831999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:14:29.6832642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6833296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.6833809Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:14:29.6834282Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:14:29.6834606Z ok (22.360s)
2022-11-23T02:14:29.6834758Z 
2022-11-23T02:14:29.6835024Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6835405Z Ran 1 test in 22.361s
2022-11-23T02:14:29.6835572Z 
2022-11-23T02:14:29.6835665Z OK
2022-11-23T02:14:29.6835781Z 
2022-11-23T02:14:29.6835908Z Generating XML reports...
2022-11-23T02:14:29.6836511Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020806.xml
2022-11-23T02:14:29.6837214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6837640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6838203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6838668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6838892Z 
2022-11-23T02:14:29.6839003Z Running tests...
2022-11-23T02:14:29.6839387Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6839912Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6840563Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.6840873Z 
2022-11-23T02:14:29.6841132Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6841434Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6841591Z 
2022-11-23T02:14:29.6841699Z OK (skipped=1)
2022-11-23T02:14:29.6841853Z 
2022-11-23T02:14:29.6841977Z Generating XML reports...
2022-11-23T02:14:29.6842556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020831.xml
2022-11-23T02:14:29.6843317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6843758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6844334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6844782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6845009Z 
2022-11-23T02:14:29.6845117Z Running tests...
2022-11-23T02:14:29.6845515Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6846020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6846648Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.6846952Z 
2022-11-23T02:14:29.6847213Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6847534Z Ran 1 test in 0.003s
2022-11-23T02:14:29.6847673Z 
2022-11-23T02:14:29.6847782Z OK (skipped=1)
2022-11-23T02:14:29.6847935Z 
2022-11-23T02:14:29.6848058Z Generating XML reports...
2022-11-23T02:14:29.6848656Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020833.xml
2022-11-23T02:14:29.6849345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6849791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6850359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6850821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6851047Z 
2022-11-23T02:14:29.6851144Z Running tests...
2022-11-23T02:14:29.6851545Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6852068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6852773Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.6853080Z 
2022-11-23T02:14:29.6853341Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6853670Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6853826Z 
2022-11-23T02:14:29.6853935Z OK (skipped=1)
2022-11-23T02:14:29.6854092Z 
2022-11-23T02:14:29.6854198Z Generating XML reports...
2022-11-23T02:14:29.6854797Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020835.xml
2022-11-23T02:14:29.6855509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6855957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6856510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6856986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6857218Z 
2022-11-23T02:14:29.6857328Z Running tests...
2022-11-23T02:14:29.6857709Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6858436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6859092Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.6859407Z 
2022-11-23T02:14:29.6859667Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6859969Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6860218Z 
2022-11-23T02:14:29.6860335Z OK (skipped=1)
2022-11-23T02:14:29.6860489Z 
2022-11-23T02:14:29.6860619Z Generating XML reports...
2022-11-23T02:14:29.6861219Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020838.xml
2022-11-23T02:14:29.6861915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6862361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6862931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6863378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6863603Z 
2022-11-23T02:14:29.6863713Z Running tests...
2022-11-23T02:14:29.6864114Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6864638Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6865280Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.6865588Z 
2022-11-23T02:14:29.6865856Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6866180Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6866343Z 
2022-11-23T02:14:29.6866451Z OK (skipped=1)
2022-11-23T02:14:29.6866587Z 
2022-11-23T02:14:29.6866709Z Generating XML reports...
2022-11-23T02:14:29.6867314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020840.xml
2022-11-23T02:14:29.6868021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6868453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6869085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6869558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6869785Z 
2022-11-23T02:14:29.6869893Z Running tests...
2022-11-23T02:14:29.6870335Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6870872Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6871395Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6871868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35384
2022-11-23T02:14:29.6872315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35385
2022-11-23T02:14:29.6872917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6873363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6873912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6874377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6874955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6875396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6875946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6876416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6876868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6877341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6878054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6878741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6879257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6879708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6880051Z ok (5.859s)
2022-11-23T02:14:29.6880202Z 
2022-11-23T02:14:29.6880467Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6880794Z Ran 1 test in 5.859s
2022-11-23T02:14:29.6880936Z 
2022-11-23T02:14:29.6881028Z OK
2022-11-23T02:14:29.6881160Z 
2022-11-23T02:14:29.6881284Z Generating XML reports...
2022-11-23T02:14:29.6881883Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020843.xml
2022-11-23T02:14:29.6882577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6883028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6883599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6884061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6884269Z 
2022-11-23T02:14:29.6884381Z Running tests...
2022-11-23T02:14:29.6884781Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6885304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6885798Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6886300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35498
2022-11-23T02:14:29.6886745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35499
2022-11-23T02:14:29.6887408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6887847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6888418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6888885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6889455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6889875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6890441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6890906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6891338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6891834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6892484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6893220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6893706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6894177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6894550Z ok (5.843s)
2022-11-23T02:14:29.6894752Z 
2022-11-23T02:14:29.6895026Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6895333Z Ran 1 test in 5.843s
2022-11-23T02:14:29.6895500Z 
2022-11-23T02:14:29.6895591Z OK
2022-11-23T02:14:29.6895724Z 
2022-11-23T02:14:29.6895849Z Generating XML reports...
2022-11-23T02:14:29.6896439Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020851.xml
2022-11-23T02:14:29.6897142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6897591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6898388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6898840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6899068Z 
2022-11-23T02:14:29.6899182Z Running tests...
2022-11-23T02:14:29.6899590Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6900095Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6900620Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6901115Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35612
2022-11-23T02:14:29.6901561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35613
2022-11-23T02:14:29.6902150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6902588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6903156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6903604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6904181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6904622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6905256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6905713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6906165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6906666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6907070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6907466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6907700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6907906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6908013Z ok (6.880s)
2022-11-23T02:14:29.6908032Z 
2022-11-23T02:14:29.6908300Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6908412Z Ran 1 test in 6.880s
2022-11-23T02:14:29.6908431Z 
2022-11-23T02:14:29.6908521Z OK
2022-11-23T02:14:29.6908540Z 
2022-11-23T02:14:29.6908669Z Generating XML reports...
2022-11-23T02:14:29.6909120Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020859.xml
2022-11-23T02:14:29.6909492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6909648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6910096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6910284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6910306Z 
2022-11-23T02:14:29.6910415Z Running tests...
2022-11-23T02:14:29.6910675Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6910992Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6911257Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6911478Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35727
2022-11-23T02:14:29.6911694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35728
2022-11-23T02:14:29.6912043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6912226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6912601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6912798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6913159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6913333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6913707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6913894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6914118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6914362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6914760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6915203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6915437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6915668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6915772Z ok (5.833s)
2022-11-23T02:14:29.6915791Z 
2022-11-23T02:14:29.6916055Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6916168Z Ran 1 test in 5.833s
2022-11-23T02:14:29.6916187Z 
2022-11-23T02:14:29.6916260Z OK
2022-11-23T02:14:29.6916279Z 
2022-11-23T02:14:29.6916418Z Generating XML reports...
2022-11-23T02:14:29.6916863Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020909.xml
2022-11-23T02:14:29.6917238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6917423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6917794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6917987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6918006Z 
2022-11-23T02:14:29.6918117Z Running tests...
2022-11-23T02:14:29.6918375Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6918669Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6918939Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6919750Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s)
2022-11-23T02:14:29.6919771Z 
2022-11-23T02:14:29.6920035Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6920147Z Ran 1 test in 1.648s
2022-11-23T02:14:29.6920166Z 
2022-11-23T02:14:29.6920273Z OK (skipped=1)
2022-11-23T02:14:29.6920292Z 
2022-11-23T02:14:29.6920417Z Generating XML reports...
2022-11-23T02:14:29.6920861Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020917.xml
2022-11-23T02:14:29.6921231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6921415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6921775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6921967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6921986Z 
2022-11-23T02:14:29.6922099Z Running tests...
2022-11-23T02:14:29.6922363Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6922678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6922925Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:14:29.6922945Z 
2022-11-23T02:14:29.6923201Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6923310Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6923329Z 
2022-11-23T02:14:29.6923422Z OK (skipped=1)
2022-11-23T02:14:29.6923459Z 
2022-11-23T02:14:29.6923564Z Generating XML reports...
2022-11-23T02:14:29.6924006Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020921.xml
2022-11-23T02:14:29.6924430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6924612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6924991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6925182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6925201Z 
2022-11-23T02:14:29.6925317Z Running tests...
2022-11-23T02:14:29.6925578Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6925923Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6926202Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:14:29.6926222Z 
2022-11-23T02:14:29.6926478Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6926591Z Ran 1 test in 0.003s
2022-11-23T02:14:29.6926609Z 
2022-11-23T02:14:29.6926717Z OK (skipped=1)
2022-11-23T02:14:29.6926736Z 
2022-11-23T02:14:29.6926862Z Generating XML reports...
2022-11-23T02:14:29.6927313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020924.xml
2022-11-23T02:14:29.6927680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6927855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6928213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6928460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6928480Z 
2022-11-23T02:14:29.6928589Z Running tests...
2022-11-23T02:14:29.6928857Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6929169Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6929481Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:14:29.6929501Z 
2022-11-23T02:14:29.6929760Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6929871Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6929890Z 
2022-11-23T02:14:29.6929999Z OK (skipped=1)
2022-11-23T02:14:29.6930018Z 
2022-11-23T02:14:29.6930122Z Generating XML reports...
2022-11-23T02:14:29.6930571Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020926.xml
2022-11-23T02:14:29.6930940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6931121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6931494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6931685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6931704Z 
2022-11-23T02:14:29.6931813Z Running tests...
2022-11-23T02:14:29.6932076Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6932385Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6932666Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6932889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35974
2022-11-23T02:14:29.6933107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35975
2022-11-23T02:14:29.6933562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6933741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6934122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6934312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6934677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6934829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6935203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6935396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6935643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6935885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6936282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6936674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6936905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6937131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6937265Z ok (4.251s)
2022-11-23T02:14:29.6937285Z 
2022-11-23T02:14:29.6937550Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6937665Z Ran 1 test in 4.252s
2022-11-23T02:14:29.6937685Z 
2022-11-23T02:14:29.6937778Z OK
2022-11-23T02:14:29.6937797Z 
2022-11-23T02:14:29.6938109Z Generating XML reports...
2022-11-23T02:14:29.6938574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020929.xml
2022-11-23T02:14:29.6938952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6939121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6939495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6939667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6939687Z 
2022-11-23T02:14:29.6939805Z Running tests...
2022-11-23T02:14:29.6940072Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6940382Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6940681Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6940899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36079
2022-11-23T02:14:29.6941113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36080
2022-11-23T02:14:29.6941486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6941642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6942023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6942216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6942581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6942827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6943214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6943403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6943646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6943892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6944270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6944673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6944900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6945130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6945237Z ok (4.224s)
2022-11-23T02:14:29.6945257Z 
2022-11-23T02:14:29.6945519Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6945632Z Ran 1 test in 4.225s
2022-11-23T02:14:29.6945651Z 
2022-11-23T02:14:29.6945743Z OK
2022-11-23T02:14:29.6945762Z 
2022-11-23T02:14:29.6945891Z Generating XML reports...
2022-11-23T02:14:29.6946319Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020935.xml
2022-11-23T02:14:29.6946688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6946932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6947312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6947511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6947531Z 
2022-11-23T02:14:29.6947640Z Running tests...
2022-11-23T02:14:29.6947901Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6948220Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6948476Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:14:29.6948514Z 
2022-11-23T02:14:29.6948756Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6948873Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6948892Z 
2022-11-23T02:14:29.6949003Z OK (skipped=1)
2022-11-23T02:14:29.6949022Z 
2022-11-23T02:14:29.6949145Z Generating XML reports...
2022-11-23T02:14:29.6949598Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020942.xml
2022-11-23T02:14:29.6949965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6950140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6950519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6950688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6950726Z 
2022-11-23T02:14:29.6950817Z Running tests...
2022-11-23T02:14:29.6951083Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6951396Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6951695Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:14:29.6951715Z 
2022-11-23T02:14:29.6952023Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6952142Z Ran 1 test in 0.002s
2022-11-23T02:14:29.6952162Z 
2022-11-23T02:14:29.6952272Z OK (skipped=1)
2022-11-23T02:14:29.6952291Z 
2022-11-23T02:14:29.6952416Z Generating XML reports...
2022-11-23T02:14:29.6952845Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020944.xml
2022-11-23T02:14:29.6953218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6953394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6953778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6953972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6953991Z 
2022-11-23T02:14:29.6954100Z Running tests...
2022-11-23T02:14:29.6954369Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6954680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6954958Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6955710Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s)
2022-11-23T02:14:29.6955732Z 
2022-11-23T02:14:29.6955992Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6956136Z Ran 1 test in 1.658s
2022-11-23T02:14:29.6956155Z 
2022-11-23T02:14:29.6956267Z OK (skipped=1)
2022-11-23T02:14:29.6956286Z 
2022-11-23T02:14:29.6956418Z Generating XML reports...
2022-11-23T02:14:29.6956872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020947.xml
2022-11-23T02:14:29.6957240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6957415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6957793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6957982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6958002Z 
2022-11-23T02:14:29.6958138Z Running tests...
2022-11-23T02:14:29.6958382Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6958701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6958984Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6959205Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36284
2022-11-23T02:14:29.6959421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36285
2022-11-23T02:14:29.6959790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6959964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6960347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6960514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6960882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6961056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6961490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6961688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6961935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6962184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6962588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6962984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6963199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6963426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6963685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgzj7jxv
2022-11-23T02:14:29.6963956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgzj7jxv/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6964206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzu59u4zf
2022-11-23T02:14:29.6964476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzu59u4zf/_remote_module_non_scriptable.py
2022-11-23T02:14:29.6964578Z ok (7.471s)
2022-11-23T02:14:29.6964598Z 
2022-11-23T02:14:29.6964862Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6964957Z Ran 1 test in 7.471s
2022-11-23T02:14:29.6965046Z 
2022-11-23T02:14:29.6965123Z OK
2022-11-23T02:14:29.6965142Z 
2022-11-23T02:14:29.6965271Z Generating XML reports...
2022-11-23T02:14:29.6965724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020951.xml
2022-11-23T02:14:29.6966098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6966274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6966657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6966848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6966867Z 
2022-11-23T02:14:29.6966981Z Running tests...
2022-11-23T02:14:29.6967224Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6967536Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6967806Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6968025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36403
2022-11-23T02:14:29.6968247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36404
2022-11-23T02:14:29.6968614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6968794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6969176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6969369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6969715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6969895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6970268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6970497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6970750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6970993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6971398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6971796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6972026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6972238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6972344Z ok (6.925s)
2022-11-23T02:14:29.6972363Z 
2022-11-23T02:14:29.6972639Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6972754Z Ran 1 test in 6.925s
2022-11-23T02:14:29.6972773Z 
2022-11-23T02:14:29.6972866Z OK
2022-11-23T02:14:29.6972885Z 
2022-11-23T02:14:29.6973011Z Generating XML reports...
2022-11-23T02:14:29.6973462Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021001.xml
2022-11-23T02:14:29.6973833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6973988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6974368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6974607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6974627Z 
2022-11-23T02:14:29.6974738Z Running tests...
2022-11-23T02:14:29.6975003Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6975318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6975611Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6975830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36518
2022-11-23T02:14:29.6976044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36519
2022-11-23T02:14:29.6976395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6976576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6976959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6977149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6977514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6977690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6978234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6978428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6978661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6978899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6979306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6979706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6980006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6980244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6980349Z ok (6.943s)
2022-11-23T02:14:29.6980369Z 
2022-11-23T02:14:29.6980639Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6980752Z Ran 1 test in 6.944s
2022-11-23T02:14:29.6980771Z 
2022-11-23T02:14:29.6980845Z OK
2022-11-23T02:14:29.6980889Z 
2022-11-23T02:14:29.6980995Z Generating XML reports...
2022-11-23T02:14:29.6981448Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021011.xml
2022-11-23T02:14:29.6981826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6982002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6982385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6982576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6982595Z 
2022-11-23T02:14:29.6982707Z Running tests...
2022-11-23T02:14:29.6982969Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6983263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6983542Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6984291Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2022-11-23T02:14:29.6984370Z 
2022-11-23T02:14:29.6984647Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6984763Z Ran 1 test in 1.621s
2022-11-23T02:14:29.6984782Z 
2022-11-23T02:14:29.6984891Z OK (skipped=1)
2022-11-23T02:14:29.6984910Z 
2022-11-23T02:14:29.6985037Z Generating XML reports...
2022-11-23T02:14:29.6985489Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021020.xml
2022-11-23T02:14:29.6985858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6986036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6986396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6986588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6986608Z 
2022-11-23T02:14:29.6986723Z Running tests...
2022-11-23T02:14:29.6986988Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6987299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6987597Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6988335Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2022-11-23T02:14:29.6988358Z 
2022-11-23T02:14:29.6988616Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6988728Z Ran 1 test in 1.646s
2022-11-23T02:14:29.6988747Z 
2022-11-23T02:14:29.6988854Z OK (skipped=1)
2022-11-23T02:14:29.6988873Z 
2022-11-23T02:14:29.6988977Z Generating XML reports...
2022-11-23T02:14:29.6989476Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021024.xml
2022-11-23T02:14:29.6989854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6990031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6990410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6990603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6990622Z 
2022-11-23T02:14:29.6990736Z Running tests...
2022-11-23T02:14:29.6991000Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6991295Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6991608Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6991830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36701
2022-11-23T02:14:29.6992048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36702
2022-11-23T02:14:29.6992417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6992589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6993008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6993253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6993621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6993778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6994149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6994333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6994580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.6994823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.6995221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6995619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.6995851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.6996084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.6996215Z skip: Need at least 4 CUDA devices (4.113s)
2022-11-23T02:14:29.6996235Z 
2022-11-23T02:14:29.6996500Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6996620Z Ran 1 test in 4.114s
2022-11-23T02:14:29.6996638Z 
2022-11-23T02:14:29.6996749Z OK (skipped=1)
2022-11-23T02:14:29.6996768Z 
2022-11-23T02:14:29.6996894Z Generating XML reports...
2022-11-23T02:14:29.6997340Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021028.xml
2022-11-23T02:14:29.6997713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.6997894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.6998273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.6998489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.6998510Z 
2022-11-23T02:14:29.6998625Z Running tests...
2022-11-23T02:14:29.6998892Z ----------------------------------------------------------------------
2022-11-23T02:14:29.6999207Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.6999532Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.6999756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36806
2022-11-23T02:14:29.6999969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36807
2022-11-23T02:14:29.7000345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7000515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7000875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7001064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7001429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7001597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7001971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7002155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7002471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7002715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7003098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7003493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7003723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7003948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7004102Z skip: Need at least 4 CUDA devices (4.276s)
2022-11-23T02:14:29.7004121Z 
2022-11-23T02:14:29.7004382Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7004502Z Ran 1 test in 4.276s
2022-11-23T02:14:29.7004521Z 
2022-11-23T02:14:29.7004632Z OK (skipped=1)
2022-11-23T02:14:29.7004650Z 
2022-11-23T02:14:29.7004775Z Generating XML reports...
2022-11-23T02:14:29.7005206Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021035.xml
2022-11-23T02:14:29.7005575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7005747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7006121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7006311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7006330Z 
2022-11-23T02:14:29.7006439Z Running tests...
2022-11-23T02:14:29.7006699Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7007016Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7007295Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7008065Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.611s)
2022-11-23T02:14:29.7008108Z 
2022-11-23T02:14:29.7008356Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7008473Z Ran 1 test in 1.611s
2022-11-23T02:14:29.7008491Z 
2022-11-23T02:14:29.7008601Z OK (skipped=1)
2022-11-23T02:14:29.7008620Z 
2022-11-23T02:14:29.7008742Z Generating XML reports...
2022-11-23T02:14:29.7009185Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021042.xml
2022-11-23T02:14:29.7009559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7009735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7010113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7010301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7010321Z 
2022-11-23T02:14:29.7010411Z Running tests...
2022-11-23T02:14:29.7010670Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7010980Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7011243Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7011313Z 
2022-11-23T02:14:29.7011582Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7011696Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7011716Z 
2022-11-23T02:14:29.7011826Z OK (skipped=1)
2022-11-23T02:14:29.7011845Z 
2022-11-23T02:14:29.7011973Z Generating XML reports...
2022-11-23T02:14:29.7012400Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021046.xml
2022-11-23T02:14:29.7012765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7012936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7013306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7013493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7013512Z 
2022-11-23T02:14:29.7013626Z Running tests...
2022-11-23T02:14:29.7013887Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7014195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7014459Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7014479Z 
2022-11-23T02:14:29.7014719Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7014831Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7014850Z 
2022-11-23T02:14:29.7014956Z OK (skipped=1)
2022-11-23T02:14:29.7014975Z 
2022-11-23T02:14:29.7015096Z Generating XML reports...
2022-11-23T02:14:29.7015540Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021048.xml
2022-11-23T02:14:29.7015910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7016084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7016457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7016690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7016711Z 
2022-11-23T02:14:29.7016806Z Running tests...
2022-11-23T02:14:29.7017068Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7017376Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7017650Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7017669Z 
2022-11-23T02:14:29.7018158Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7018274Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7018293Z 
2022-11-23T02:14:29.7018408Z OK (skipped=1)
2022-11-23T02:14:29.7018427Z 
2022-11-23T02:14:29.7018553Z Generating XML reports...
2022-11-23T02:14:29.7019002Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021051.xml
2022-11-23T02:14:29.7019351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7019525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7019896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7020082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7020101Z 
2022-11-23T02:14:29.7020211Z Running tests...
2022-11-23T02:14:29.7020473Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7020781Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7021127Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7021146Z 
2022-11-23T02:14:29.7021410Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7021508Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7021527Z 
2022-11-23T02:14:29.7021636Z OK (skipped=1)
2022-11-23T02:14:29.7021655Z 
2022-11-23T02:14:29.7021780Z Generating XML reports...
2022-11-23T02:14:29.7022222Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021053.xml
2022-11-23T02:14:29.7022587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7022759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7023133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7023325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7023345Z 
2022-11-23T02:14:29.7023435Z Running tests...
2022-11-23T02:14:29.7023694Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7024008Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7024268Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7024288Z 
2022-11-23T02:14:29.7024542Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7024652Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7024671Z 
2022-11-23T02:14:29.7024775Z OK (skipped=1)
2022-11-23T02:14:29.7024794Z 
2022-11-23T02:14:29.7024915Z Generating XML reports...
2022-11-23T02:14:29.7025363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021056.xml
2022-11-23T02:14:29.7025717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7025896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7026346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7026674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7026698Z 
2022-11-23T02:14:29.7026870Z Running tests...
2022-11-23T02:14:29.7027265Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7027588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7027850Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7027870Z 
2022-11-23T02:14:29.7028131Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7028225Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7028244Z 
2022-11-23T02:14:29.7028352Z OK (skipped=1)
2022-11-23T02:14:29.7028372Z 
2022-11-23T02:14:29.7028495Z Generating XML reports...
2022-11-23T02:14:29.7028946Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021058.xml
2022-11-23T02:14:29.7029313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7029488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7029863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7030051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7030071Z 
2022-11-23T02:14:29.7030178Z Running tests...
2022-11-23T02:14:29.7030486Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7030794Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7031063Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7031082Z 
2022-11-23T02:14:29.7031340Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7031452Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7031471Z 
2022-11-23T02:14:29.7031578Z OK (skipped=1)
2022-11-23T02:14:29.7031596Z 
2022-11-23T02:14:29.7031720Z Generating XML reports...
2022-11-23T02:14:29.7032165Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021100.xml
2022-11-23T02:14:29.7032513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7032693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7033066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7033253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7033275Z 
2022-11-23T02:14:29.7033383Z Running tests...
2022-11-23T02:14:29.7033642Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7033951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7034211Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7034231Z 
2022-11-23T02:14:29.7034485Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7034578Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7034597Z 
2022-11-23T02:14:29.7034709Z OK (skipped=1)
2022-11-23T02:14:29.7034732Z 
2022-11-23T02:14:29.7034854Z Generating XML reports...
2022-11-23T02:14:29.7035301Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021103.xml
2022-11-23T02:14:29.7035718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7035899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7036277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7036465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7036485Z 
2022-11-23T02:14:29.7036593Z Running tests...
2022-11-23T02:14:29.7036835Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7037144Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7037399Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7037418Z 
2022-11-23T02:14:29.7037675Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7037787Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7037809Z 
2022-11-23T02:14:29.7037918Z OK (skipped=1)
2022-11-23T02:14:29.7037936Z 
2022-11-23T02:14:29.7038059Z Generating XML reports...
2022-11-23T02:14:29.7038498Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021105.xml
2022-11-23T02:14:29.7038845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7039020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7039392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7039632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7039652Z 
2022-11-23T02:14:29.7039765Z Running tests...
2022-11-23T02:14:29.7040026Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7040341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7040596Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7040616Z 
2022-11-23T02:14:29.7040876Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7040969Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7040988Z 
2022-11-23T02:14:29.7041096Z OK (skipped=1)
2022-11-23T02:14:29.7041115Z 
2022-11-23T02:14:29.7041237Z Generating XML reports...
2022-11-23T02:14:29.7041678Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021108.xml
2022-11-23T02:14:29.7042050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7042224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7042605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7042796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7042815Z 
2022-11-23T02:14:29.7042923Z Running tests...
2022-11-23T02:14:29.7043166Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7043476Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7043728Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7043947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37275
2022-11-23T02:14:29.7044168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37276
2022-11-23T02:14:29.7044535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7044752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7045136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7045305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7045665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7045838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7046205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7046397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7046646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7046888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7047290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7047684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7047895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7048120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7048454Z STAGE:2022-11-23 02:11:16 37275:37275 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7049212Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:14:29.7049376Z   warnings.warn(
2022-11-23T02:14:29.7049715Z STAGE:2022-11-23 02:11:16 37276:37276 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7050468Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:14:29.7050581Z   warnings.warn(
2022-11-23T02:14:29.7050916Z STAGE:2022-11-23 02:11:16 37275:37275 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7051247Z STAGE:2022-11-23 02:11:16 37276:37276 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7051581Z STAGE:2022-11-23 02:11:16 37275:37275 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7051925Z STAGE:2022-11-23 02:11:16 37276:37276 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7052251Z STAGE:2022-11-23 02:11:16 37275:37275 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7052577Z STAGE:2022-11-23 02:11:16 37276:37276 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7052908Z STAGE:2022-11-23 02:11:16 37275:37275 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7053252Z STAGE:2022-11-23 02:11:16 37275:37275 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7053582Z STAGE:2022-11-23 02:11:16 37276:37276 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7053924Z STAGE:2022-11-23 02:11:16 37276:37276 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7054013Z ok (6.388s)
2022-11-23T02:14:29.7054050Z 
2022-11-23T02:14:29.7054294Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7054408Z Ran 1 test in 6.388s
2022-11-23T02:14:29.7054479Z 
2022-11-23T02:14:29.7054576Z OK
2022-11-23T02:14:29.7054596Z 
2022-11-23T02:14:29.7054722Z Generating XML reports...
2022-11-23T02:14:29.7055171Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021110.xml
2022-11-23T02:14:29.7055535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7055712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7056092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7056265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7056285Z 
2022-11-23T02:14:29.7056394Z Running tests...
2022-11-23T02:14:29.7056652Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7056967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7057225Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7057245Z 
2022-11-23T02:14:29.7057500Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7057613Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7057632Z 
2022-11-23T02:14:29.7057742Z OK (skipped=1)
2022-11-23T02:14:29.7057761Z 
2022-11-23T02:14:29.7058104Z Generating XML reports...
2022-11-23T02:14:29.7058551Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021119.xml
2022-11-23T02:14:29.7059003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7059177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7059554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7059745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7059765Z 
2022-11-23T02:14:29.7059875Z Running tests...
2022-11-23T02:14:29.7060132Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7060444Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7060696Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7060915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37426
2022-11-23T02:14:29.7061135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37427
2022-11-23T02:14:29.7061505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7061685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7062058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7062248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7062606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7062777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7063130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7063318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7063566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7063808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7064262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7064668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7064899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7065126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7065459Z STAGE:2022-11-23 02:11:27 37427:37427 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7065763Z STAGE:2022-11-23 02:11:27 37426:37426 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7066100Z STAGE:2022-11-23 02:11:28 37426:37426 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7066432Z STAGE:2022-11-23 02:11:28 37427:37427 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7066777Z STAGE:2022-11-23 02:11:28 37426:37426 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7067118Z STAGE:2022-11-23 02:11:28 37427:37427 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7067443Z STAGE:2022-11-23 02:11:28 37426:37426 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7067774Z STAGE:2022-11-23 02:11:28 37426:37426 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7068117Z STAGE:2022-11-23 02:11:28 37426:37426 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7068498Z STAGE:2022-11-23 02:11:28 37427:37427 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7068808Z STAGE:2022-11-23 02:11:28 37427:37427 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7069151Z STAGE:2022-11-23 02:11:28 37427:37427 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7069255Z ok (7.762s)
2022-11-23T02:14:29.7069274Z 
2022-11-23T02:14:29.7069539Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7069651Z Ran 1 test in 7.763s
2022-11-23T02:14:29.7069670Z 
2022-11-23T02:14:29.7069763Z OK
2022-11-23T02:14:29.7069781Z 
2022-11-23T02:14:29.7069909Z Generating XML reports...
2022-11-23T02:14:29.7070357Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021121.xml
2022-11-23T02:14:29.7070708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7070888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7071264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7071457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7071476Z 
2022-11-23T02:14:29.7071584Z Running tests...
2022-11-23T02:14:29.7071843Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7072155Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7072418Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7072635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37545
2022-11-23T02:14:29.7072834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37546
2022-11-23T02:14:29.7073207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7073380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7073803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7073998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7074359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7074533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7074903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7075072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7075319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7075562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7075967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7076364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7076591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7076819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7076923Z ok (6.983s)
2022-11-23T02:14:29.7076943Z 
2022-11-23T02:14:29.7077203Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7077298Z Ran 1 test in 6.983s
2022-11-23T02:14:29.7077317Z 
2022-11-23T02:14:29.7077490Z OK
2022-11-23T02:14:29.7077510Z 
2022-11-23T02:14:29.7077636Z Generating XML reports...
2022-11-23T02:14:29.7078078Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021132.xml
2022-11-23T02:14:29.7078450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7078626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7079000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7079190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7079210Z 
2022-11-23T02:14:29.7079301Z Running tests...
2022-11-23T02:14:29.7079562Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7079878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7080132Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7080152Z 
2022-11-23T02:14:29.7080411Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7080528Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7080547Z 
2022-11-23T02:14:29.7080659Z OK (skipped=1)
2022-11-23T02:14:29.7080678Z 
2022-11-23T02:14:29.7080801Z Generating XML reports...
2022-11-23T02:14:29.7081247Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021141.xml
2022-11-23T02:14:29.7081594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7081771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7082146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7082336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7082356Z 
2022-11-23T02:14:29.7082464Z Running tests...
2022-11-23T02:14:29.7082722Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7083078Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7083336Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7083553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37693
2022-11-23T02:14:29.7083752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37694
2022-11-23T02:14:29.7084124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7084296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7084673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7084861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7085223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7085396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7085764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7085932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7086175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7086417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7086813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7087254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7087485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7087708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7088042Z STAGE:2022-11-23 02:11:49 37694:37694 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7088366Z STAGE:2022-11-23 02:11:49 37693:37693 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7088681Z STAGE:2022-11-23 02:11:49 37693:37693 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7089008Z STAGE:2022-11-23 02:11:49 37694:37694 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7089354Z STAGE:2022-11-23 02:11:49 37693:37693 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7089699Z STAGE:2022-11-23 02:11:49 37694:37694 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7090023Z STAGE:2022-11-23 02:11:49 37694:37694 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7090353Z STAGE:2022-11-23 02:11:49 37694:37694 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7090697Z STAGE:2022-11-23 02:11:49 37694:37694 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7091017Z STAGE:2022-11-23 02:11:49 37693:37693 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7091342Z STAGE:2022-11-23 02:11:49 37693:37693 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7091664Z STAGE:2022-11-23 02:11:49 37693:37693 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7091771Z ok (6.376s)
2022-11-23T02:14:29.7091790Z 
2022-11-23T02:14:29.7092049Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7092161Z Ran 1 test in 6.376s
2022-11-23T02:14:29.7092180Z 
2022-11-23T02:14:29.7092272Z OK
2022-11-23T02:14:29.7092348Z 
2022-11-23T02:14:29.7092477Z Generating XML reports...
2022-11-23T02:14:29.7092933Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021144.xml
2022-11-23T02:14:29.7093344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7093502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7093877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7094066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7094090Z 
2022-11-23T02:14:29.7094201Z Running tests...
2022-11-23T02:14:29.7094461Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7094771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7095035Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7095255Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37811
2022-11-23T02:14:29.7095468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37812
2022-11-23T02:14:29.7095818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7095992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7096361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7096606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7096968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7097147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7097516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7097705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7098138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7098387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7098785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7099184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7099418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7099643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7100130Z STAGE:2022-11-23 02:11:58 37811:37811 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7100457Z STAGE:2022-11-23 02:11:58 37812:37812 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7100796Z STAGE:2022-11-23 02:11:58 37812:37812 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7101101Z STAGE:2022-11-23 02:11:58 37811:37811 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7101665Z STAGE:2022-11-23 02:11:58 37812:37812 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:11:58 37811:37811 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7101692Z 
2022-11-23T02:14:29.7102111Z STAGE:2022-11-23 02:11:58 37812:37812 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7102509Z STAGE:2022-11-23 02:11:58 37811:37811 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7102853Z STAGE:2022-11-23 02:11:58 37812:37812 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7103198Z STAGE:2022-11-23 02:11:58 37812:37812 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7103526Z STAGE:2022-11-23 02:11:58 37811:37811 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7103865Z STAGE:2022-11-23 02:11:58 37811:37811 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7103971Z ok (6.360s)
2022-11-23T02:14:29.7103994Z 
2022-11-23T02:14:29.7104259Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7104353Z Ran 1 test in 6.361s
2022-11-23T02:14:29.7104373Z 
2022-11-23T02:14:29.7104465Z OK
2022-11-23T02:14:29.7104484Z 
2022-11-23T02:14:29.7104607Z Generating XML reports...
2022-11-23T02:14:29.7105064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021152.xml
2022-11-23T02:14:29.7105427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7105603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7105977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7106165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7106185Z 
2022-11-23T02:14:29.7106295Z Running tests...
2022-11-23T02:14:29.7106610Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7106917Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7107169Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7107188Z 
2022-11-23T02:14:29.7107430Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7107531Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7107550Z 
2022-11-23T02:14:29.7107648Z OK (skipped=1)
2022-11-23T02:14:29.7107667Z 
2022-11-23T02:14:29.7107791Z Generating XML reports...
2022-11-23T02:14:29.7108236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021201.xml
2022-11-23T02:14:29.7108585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7108763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7109135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7109324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7109346Z 
2022-11-23T02:14:29.7109456Z Running tests...
2022-11-23T02:14:29.7109714Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7110025Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7110275Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7110295Z 
2022-11-23T02:14:29.7110553Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7110647Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7110667Z 
2022-11-23T02:14:29.7110775Z OK (skipped=1)
2022-11-23T02:14:29.7110797Z 
2022-11-23T02:14:29.7110921Z Generating XML reports...
2022-11-23T02:14:29.7111363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021204.xml
2022-11-23T02:14:29.7111774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7111953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7112335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7112524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7112543Z 
2022-11-23T02:14:29.7112652Z Running tests...
2022-11-23T02:14:29.7112893Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7113203Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7113466Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7113485Z 
2022-11-23T02:14:29.7113740Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7113850Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7113872Z 
2022-11-23T02:14:29.7113982Z OK (skipped=1)
2022-11-23T02:14:29.7114001Z 
2022-11-23T02:14:29.7114124Z Generating XML reports...
2022-11-23T02:14:29.7114570Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021206.xml
2022-11-23T02:14:29.7114918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7115095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7115469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7115711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7115731Z 
2022-11-23T02:14:29.7115841Z Running tests...
2022-11-23T02:14:29.7116103Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7116418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7116681Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7116700Z 
2022-11-23T02:14:29.7116957Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7117050Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7117069Z 
2022-11-23T02:14:29.7117177Z OK (skipped=1)
2022-11-23T02:14:29.7117196Z 
2022-11-23T02:14:29.7117318Z Generating XML reports...
2022-11-23T02:14:29.7117761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021209.xml
2022-11-23T02:14:29.7118130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7118306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7118687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7118875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7118894Z 
2022-11-23T02:14:29.7119004Z Running tests...
2022-11-23T02:14:29.7119247Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7119558Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7119808Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7120026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38061
2022-11-23T02:14:29.7120247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38062
2022-11-23T02:14:29.7120615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7120833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7121215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7121386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7121748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7121921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7122295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7122481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7122726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7122969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7123369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7123760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7123969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7124196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7124528Z STAGE:2022-11-23 02:12:17 38061:38061 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7124848Z STAGE:2022-11-23 02:12:17 38062:38062 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7125236Z STAGE:2022-11-23 02:12:18 38061:38061 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7125586Z STAGE:2022-11-23 02:12:18 38061:38061 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7125918Z STAGE:2022-11-23 02:12:18 38062:38062 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7126260Z STAGE:2022-11-23 02:12:18 38062:38062 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7126588Z STAGE:2022-11-23 02:12:18 38061:38061 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7126889Z STAGE:2022-11-23 02:12:18 38062:38062 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7127221Z STAGE:2022-11-23 02:12:18 38062:38062 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7127567Z STAGE:2022-11-23 02:12:18 38062:38062 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7127896Z STAGE:2022-11-23 02:12:18 38061:38061 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7128241Z STAGE:2022-11-23 02:12:18 38061:38061 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7128345Z ok (7.832s)
2022-11-23T02:14:29.7128365Z 
2022-11-23T02:14:29.7128631Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7128743Z Ran 1 test in 7.832s
2022-11-23T02:14:29.7128763Z 
2022-11-23T02:14:29.7128838Z OK
2022-11-23T02:14:29.7128873Z 
2022-11-23T02:14:29.7128978Z Generating XML reports...
2022-11-23T02:14:29.7129427Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021211.xml
2022-11-23T02:14:29.7129792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7129968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7130343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7130580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7130601Z 
2022-11-23T02:14:29.7130715Z Running tests...
2022-11-23T02:14:29.7130977Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7131270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7131534Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7131752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38184
2022-11-23T02:14:29.7131964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38185
2022-11-23T02:14:29.7132335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7132509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7132886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7133075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7133418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7133588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7133953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7134137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7134378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7134666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7135069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7135462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7135691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7135898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7136231Z STAGE:2022-11-23 02:12:27 38184:38184 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7136551Z STAGE:2022-11-23 02:12:27 38185:38185 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7136885Z STAGE:2022-11-23 02:12:28 38184:38184 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7137233Z STAGE:2022-11-23 02:12:28 38184:38184 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7137559Z STAGE:2022-11-23 02:12:28 38185:38185 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7138072Z STAGE:2022-11-23 02:12:28 38185:38185 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7138410Z STAGE:2022-11-23 02:12:29 38184:38184 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7138731Z STAGE:2022-11-23 02:12:29 38185:38185 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7139041Z STAGE:2022-11-23 02:12:29 38185:38185 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7139378Z STAGE:2022-11-23 02:12:29 38185:38185 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7139709Z STAGE:2022-11-23 02:12:29 38184:38184 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7140048Z STAGE:2022-11-23 02:12:29 38184:38184 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7140151Z ok (7.860s)
2022-11-23T02:14:29.7140237Z 
2022-11-23T02:14:29.7140509Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7140624Z Ran 1 test in 7.860s
2022-11-23T02:14:29.7140643Z 
2022-11-23T02:14:29.7140736Z OK
2022-11-23T02:14:29.7140755Z 
2022-11-23T02:14:29.7140880Z Generating XML reports...
2022-11-23T02:14:29.7141311Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021221.xml
2022-11-23T02:14:29.7141676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7141849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7142226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7142417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7142436Z 
2022-11-23T02:14:29.7142548Z Running tests...
2022-11-23T02:14:29.7142806Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7143118Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7143362Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7143400Z 
2022-11-23T02:14:29.7143643Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7143757Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7143776Z 
2022-11-23T02:14:29.7143883Z OK (skipped=1)
2022-11-23T02:14:29.7143902Z 
2022-11-23T02:14:29.7144094Z Generating XML reports...
2022-11-23T02:14:29.7144536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021232.xml
2022-11-23T02:14:29.7144904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7145076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7145451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7145622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7145659Z 
2022-11-23T02:14:29.7145750Z Running tests...
2022-11-23T02:14:29.7146009Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7146321Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7146579Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:14:29.7146602Z 
2022-11-23T02:14:29.7146861Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7146972Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7146990Z 
2022-11-23T02:14:29.7147100Z OK (skipped=1)
2022-11-23T02:14:29.7147119Z 
2022-11-23T02:14:29.7147241Z Generating XML reports...
2022-11-23T02:14:29.7147660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021234.xml
2022-11-23T02:14:29.7148020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7148191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7148561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7148748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7148771Z 
2022-11-23T02:14:29.7148880Z Running tests...
2022-11-23T02:14:29.7149138Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7149497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7149870Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:14:29.7149914Z 
2022-11-23T02:14:29.7150158Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7150482Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7150503Z 
2022-11-23T02:14:29.7150617Z OK (skipped=1)
2022-11-23T02:14:29.7150635Z 
2022-11-23T02:14:29.7150761Z Generating XML reports...
2022-11-23T02:14:29.7151220Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021237.xml
2022-11-23T02:14:29.7151587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7151765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7152143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7152328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7152348Z 
2022-11-23T02:14:29.7152455Z Running tests...
2022-11-23T02:14:29.7152698Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7153005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7153263Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.003s)
2022-11-23T02:14:29.7153283Z 
2022-11-23T02:14:29.7153538Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7153715Z Ran 1 test in 0.003s
2022-11-23T02:14:29.7153734Z 
2022-11-23T02:14:29.7153843Z OK (skipped=1)
2022-11-23T02:14:29.7153862Z 
2022-11-23T02:14:29.7153985Z Generating XML reports...
2022-11-23T02:14:29.7154440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021239.xml
2022-11-23T02:14:29.7154791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7154965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7155339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7155530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7155550Z 
2022-11-23T02:14:29.7155657Z Running tests...
2022-11-23T02:14:29.7155918Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7156234Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7156514Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T02:14:29.7156534Z 
2022-11-23T02:14:29.7156794Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7156888Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7156924Z 
2022-11-23T02:14:29.7157013Z OK (skipped=1)
2022-11-23T02:14:29.7157031Z 
2022-11-23T02:14:29.7157153Z Generating XML reports...
2022-11-23T02:14:29.7157593Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021241.xml
2022-11-23T02:14:29.7157957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7158128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7158502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7158689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7158708Z 
2022-11-23T02:14:29.7158877Z Running tests...
2022-11-23T02:14:29.7159126Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7159434Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7159738Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T02:14:29.7159758Z 
2022-11-23T02:14:29.7160012Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7160124Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7160142Z 
2022-11-23T02:14:29.7160252Z OK (skipped=1)
2022-11-23T02:14:29.7160272Z 
2022-11-23T02:14:29.7160399Z Generating XML reports...
2022-11-23T02:14:29.7160841Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021244.xml
2022-11-23T02:14:29.7161210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7161366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7161741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7161931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7161950Z 
2022-11-23T02:14:29.7162058Z Running tests...
2022-11-23T02:14:29.7162315Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7162625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7162924Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T02:14:29.7162991Z 
2022-11-23T02:14:29.7163249Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7163362Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7163384Z 
2022-11-23T02:14:29.7163473Z OK (skipped=1)
2022-11-23T02:14:29.7163491Z 
2022-11-23T02:14:29.7163615Z Generating XML reports...
2022-11-23T02:14:29.7164061Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021246.xml
2022-11-23T02:14:29.7164428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7164602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7164978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7165169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7165188Z 
2022-11-23T02:14:29.7165295Z Running tests...
2022-11-23T02:14:29.7165536Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7165851Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7166136Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:14:29.7166156Z 
2022-11-23T02:14:29.7166411Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7166522Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7166541Z 
2022-11-23T02:14:29.7166649Z OK (skipped=1)
2022-11-23T02:14:29.7166668Z 
2022-11-23T02:14:29.7166791Z Generating XML reports...
2022-11-23T02:14:29.7167231Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021249.xml
2022-11-23T02:14:29.7167599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7167754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7168172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7168363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7168382Z 
2022-11-23T02:14:29.7168490Z Running tests...
2022-11-23T02:14:29.7168750Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7169057Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7169306Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7169524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38571
2022-11-23T02:14:29.7169741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38572
2022-11-23T02:14:29.7170093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7170268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7170646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7170835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7171190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7171361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7171731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7171972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7172196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7172445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7172845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7173239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7173465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7173689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7173794Z ok (5.756s)
2022-11-23T02:14:29.7173813Z 
2022-11-23T02:14:29.7174076Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7174190Z Ran 1 test in 5.756s
2022-11-23T02:14:29.7174209Z 
2022-11-23T02:14:29.7174283Z OK
2022-11-23T02:14:29.7174302Z 
2022-11-23T02:14:29.7174424Z Generating XML reports...
2022-11-23T02:14:29.7174870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021251.xml
2022-11-23T02:14:29.7175235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7175410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7175785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7175974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7175993Z 
2022-11-23T02:14:29.7176102Z Running tests...
2022-11-23T02:14:29.7176345Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7176660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7176933Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7177193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38689
2022-11-23T02:14:29.7177414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38690
2022-11-23T02:14:29.7177784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7178125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7178514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7178707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7179055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7179228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7179601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7179786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7180027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7180265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7180660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7181052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7181356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7181564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7181902Z STAGE:2022-11-23 02:13:05 38690:38690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7182223Z STAGE:2022-11-23 02:13:05 38689:38689 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7182553Z STAGE:2022-11-23 02:13:05 38690:38690 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7182874Z STAGE:2022-11-23 02:13:05 38689:38689 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7183220Z STAGE:2022-11-23 02:13:05 38690:38690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7183561Z STAGE:2022-11-23 02:13:05 38689:38689 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7183670Z ok (6.447s)
2022-11-23T02:14:29.7183689Z 
2022-11-23T02:14:29.7183933Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7184047Z Ran 1 test in 6.447s
2022-11-23T02:14:29.7184067Z 
2022-11-23T02:14:29.7184160Z OK
2022-11-23T02:14:29.7184179Z 
2022-11-23T02:14:29.7184306Z Generating XML reports...
2022-11-23T02:14:29.7184755Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021259.xml
2022-11-23T02:14:29.7185121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7185295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7185671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7185860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7185882Z 
2022-11-23T02:14:29.7185974Z Running tests...
2022-11-23T02:14:29.7186232Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7186543Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7186872Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7187097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38811
2022-11-23T02:14:29.7187311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38812
2022-11-23T02:14:29.7187679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7187852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7188210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7188402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7188762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7188938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7189308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7189494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7189736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7189974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7190370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7190794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7191018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7191246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7191579Z STAGE:2022-11-23 02:13:14 38811:38811 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7191901Z STAGE:2022-11-23 02:13:14 38812:38812 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:14:29.7192234Z STAGE:2022-11-23 02:13:14 38812:38812 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7192567Z STAGE:2022-11-23 02:13:14 38811:38811 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:14:29.7192910Z STAGE:2022-11-23 02:13:14 38812:38812 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7193304Z STAGE:2022-11-23 02:13:14 38811:38811 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:14:29.7193389Z ok (6.453s)
2022-11-23T02:14:29.7193409Z 
2022-11-23T02:14:29.7193673Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7193789Z Ran 1 test in 6.454s
2022-11-23T02:14:29.7193808Z 
2022-11-23T02:14:29.7193901Z OK
2022-11-23T02:14:29.7193919Z 
2022-11-23T02:14:29.7194044Z Generating XML reports...
2022-11-23T02:14:29.7194494Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021308.xml
2022-11-23T02:14:29.7194863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7195038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7195397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7195587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7195606Z 
2022-11-23T02:14:29.7195716Z Running tests...
2022-11-23T02:14:29.7195977Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7196336Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7196621Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:14:29.7196641Z 
2022-11-23T02:14:29.7196899Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7197013Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7197032Z 
2022-11-23T02:14:29.7197141Z OK (skipped=1)
2022-11-23T02:14:29.7197160Z 
2022-11-23T02:14:29.7197266Z Generating XML reports...
2022-11-23T02:14:29.7197705Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021317.xml
2022-11-23T02:14:29.7198075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7198252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7198628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7198815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7198834Z 
2022-11-23T02:14:29.7198944Z Running tests...
2022-11-23T02:14:29.7199205Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7199511Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7199762Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:14:29.7199827Z 
2022-11-23T02:14:29.7200090Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7200202Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7200222Z 
2022-11-23T02:14:29.7200329Z OK (skipped=1)
2022-11-23T02:14:29.7200348Z 
2022-11-23T02:14:29.7200470Z Generating XML reports...
2022-11-23T02:14:29.7200917Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021320.xml
2022-11-23T02:14:29.7201284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7201459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7201830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7202001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7202020Z 
2022-11-23T02:14:29.7202131Z Running tests...
2022-11-23T02:14:29.7202388Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7202698Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7202997Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:14:29.7203018Z 
2022-11-23T02:14:29.7203279Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7203390Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7203409Z 
2022-11-23T02:14:29.7203515Z OK (skipped=1)
2022-11-23T02:14:29.7203534Z 
2022-11-23T02:14:29.7203656Z Generating XML reports...
2022-11-23T02:14:29.7204082Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021322.xml
2022-11-23T02:14:29.7204449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7204625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7204998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7205231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7205252Z 
2022-11-23T02:14:29.7205364Z Running tests...
2022-11-23T02:14:29.7205628Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7205939Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7206211Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:14:29.7206249Z 
2022-11-23T02:14:29.7206489Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7206599Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7206621Z 
2022-11-23T02:14:29.7206725Z OK (skipped=1)
2022-11-23T02:14:29.7206744Z 
2022-11-23T02:14:29.7206866Z Generating XML reports...
2022-11-23T02:14:29.7207306Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021324.xml
2022-11-23T02:14:29.7207672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7207845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7208217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7208386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7208422Z 
2022-11-23T02:14:29.7208512Z Running tests...
2022-11-23T02:14:29.7208770Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7209077Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7209408Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-11-23T02:14:29.7209427Z 
2022-11-23T02:14:29.7209686Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7209798Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7209817Z 
2022-11-23T02:14:29.7209921Z OK (skipped=1)
2022-11-23T02:14:29.7209939Z 
2022-11-23T02:14:29.7210062Z Generating XML reports...
2022-11-23T02:14:29.7210484Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021327.xml
2022-11-23T02:14:29.7210846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7211017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7211392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7211584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7211603Z 
2022-11-23T02:14:29.7211708Z Running tests...
2022-11-23T02:14:29.7211971Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7212283Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7212548Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-11-23T02:14:29.7212585Z 
2022-11-23T02:14:29.7212825Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7212934Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7212954Z 
2022-11-23T02:14:29.7213059Z OK (skipped=1)
2022-11-23T02:14:29.7213078Z 
2022-11-23T02:14:29.7213199Z Generating XML reports...
2022-11-23T02:14:29.7213645Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021329.xml
2022-11-23T02:14:29.7214015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7214234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7214617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7214789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7214826Z 
2022-11-23T02:14:29.7214916Z Running tests...
2022-11-23T02:14:29.7215171Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7215479Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7215742Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7215965Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39131
2022-11-23T02:14:29.7216181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39132
2022-11-23T02:14:29.7216550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7216723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7217079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7217268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7217626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7217795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7218332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7218593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7218839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7219083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7219465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7219852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7220077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7220293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7220398Z ok (7.362s)
2022-11-23T02:14:29.7220422Z 
2022-11-23T02:14:29.7220687Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7220797Z Ran 1 test in 7.363s
2022-11-23T02:14:29.7220816Z 
2022-11-23T02:14:29.7220906Z OK
2022-11-23T02:14:29.7220925Z 
2022-11-23T02:14:29.7221049Z Generating XML reports...
2022-11-23T02:14:29.7221496Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021331.xml
2022-11-23T02:14:29.7221862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7222034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7222394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7222581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7222600Z 
2022-11-23T02:14:29.7222709Z Running tests...
2022-11-23T02:14:29.7222968Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7223277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7223651Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T02:14:29.7223672Z 
2022-11-23T02:14:29.7223937Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7224049Z Ran 1 test in 0.002s
2022-11-23T02:14:29.7224068Z 
2022-11-23T02:14:29.7224158Z OK (skipped=1)
2022-11-23T02:14:29.7224192Z 
2022-11-23T02:14:29.7224297Z Generating XML reports...
2022-11-23T02:14:29.7224740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021341.xml
2022-11-23T02:14:29.7225104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7225280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7225654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7225889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7225909Z 
2022-11-23T02:14:29.7226017Z Running tests...
2022-11-23T02:14:29.7226278Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7226571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7226817Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7227034Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39283
2022-11-23T02:14:29.7227251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39284
2022-11-23T02:14:29.7227671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7227846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7228220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7228407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7228752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7228924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7229294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7229478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7229718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7229963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7230363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7230757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7230980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7231187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7231438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8e_kkaov
2022-11-23T02:14:29.7231700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8e_kkaov/_remote_module_non_scriptable.py
2022-11-23T02:14:29.7231951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkk0qotz
2022-11-23T02:14:29.7232218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkk0qotz/_remote_module_non_scriptable.py
2022-11-23T02:14:29.7232320Z ok (6.828s)
2022-11-23T02:14:29.7232339Z 
2022-11-23T02:14:29.7232648Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7232764Z Ran 1 test in 6.828s
2022-11-23T02:14:29.7232783Z 
2022-11-23T02:14:29.7232875Z OK
2022-11-23T02:14:29.7232894Z 
2022-11-23T02:14:29.7232999Z Generating XML reports...
2022-11-23T02:14:29.7233445Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021344.xml
2022-11-23T02:14:29.7233810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7233982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7234355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7234547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7234566Z 
2022-11-23T02:14:29.7234669Z Running tests...
2022-11-23T02:14:29.7234929Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7235221Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7235508Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7235726Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39398
2022-11-23T02:14:29.7235936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39399
2022-11-23T02:14:29.7236297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7236520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7236897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7237083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7237441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7237595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7237960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7238143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7238383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7238624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7239026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7239421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7239646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7239869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7240105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3199nidg
2022-11-23T02:14:29.7240367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3199nidg/_remote_module_non_scriptable.py
2022-11-23T02:14:29.7240616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdm_cycph
2022-11-23T02:14:29.7240878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdm_cycph/_remote_module_non_scriptable.py
2022-11-23T02:14:29.7241699Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:14:29.7242473Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:14:29.7242580Z ok (7.433s)
2022-11-23T02:14:29.7242600Z 
2022-11-23T02:14:29.7242872Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7242984Z Ran 1 test in 7.433s
2022-11-23T02:14:29.7243003Z 
2022-11-23T02:14:29.7243094Z OK
2022-11-23T02:14:29.7243113Z 
2022-11-23T02:14:29.7243236Z Generating XML reports...
2022-11-23T02:14:29.7243680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021353.xml
2022-11-23T02:14:29.7244029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7244202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7244577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7244809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7244829Z 
2022-11-23T02:14:29.7244936Z Running tests...
2022-11-23T02:14:29.7245197Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7245506Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7245789Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7245987Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39517
2022-11-23T02:14:29.7246200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39518
2022-11-23T02:14:29.7246566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7246737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7247114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7247296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7247656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7247824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7248190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7248360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7248599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7248836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7249232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7249668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7249900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7250114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7250350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.7250586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.7250960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.7251344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.7251579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:14:29.7251814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:14:29.7252199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.7252585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.7252839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4kbgordt
2022-11-23T02:14:29.7253102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4kbgordt/_remote_module_non_scriptable.py
2022-11-23T02:14:29.7253345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mqmnlx0
2022-11-23T02:14:29.7253641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mqmnlx0/_remote_module_non_scriptable.py
2022-11-23T02:14:29.7253741Z ok (11.266s)
2022-11-23T02:14:29.7253761Z 
2022-11-23T02:14:29.7254024Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7254138Z Ran 1 test in 11.266s
2022-11-23T02:14:29.7254157Z 
2022-11-23T02:14:29.7254249Z OK
2022-11-23T02:14:29.7254267Z 
2022-11-23T02:14:29.7254389Z Generating XML reports...
2022-11-23T02:14:29.7254834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021403.xml
2022-11-23T02:14:29.7255201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7255358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7255735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7255925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7255944Z 
2022-11-23T02:14:29.7256052Z Running tests...
2022-11-23T02:14:29.7256309Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7256623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:14:29.7256907Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:29.7257124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39639
2022-11-23T02:14:29.7257337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39640
2022-11-23T02:14:29.7257686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7257858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7258405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7258595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7259022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:29.7259198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:29.7259570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:29.7259759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:29.7259983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:14:29.7260222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:29.7260621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7261008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:14:29.7261235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:14:29.7261464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:29.7261701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:14:29.7261936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:14:29.7262323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.7262537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:14:29.7262982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:14:29.7263217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:14:29.7263604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.7263991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:14:29.7264246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxachgq0u
2022-11-23T02:14:29.7264514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxachgq0u/_remote_module_non_scriptable.py
2022-11-23T02:14:29.7264764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwran3jm
2022-11-23T02:14:29.7265031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwran3jm/_remote_module_non_scriptable.py
2022-11-23T02:14:29.7265118Z ok (11.351s)
2022-11-23T02:14:29.7265153Z 
2022-11-23T02:14:29.7265400Z ----------------------------------------------------------------------
2022-11-23T02:14:29.7265514Z Ran 1 test in 11.351s
2022-11-23T02:14:29.7265533Z 
2022-11-23T02:14:29.7265623Z OK
2022-11-23T02:14:29.7265642Z 
2022-11-23T02:14:29.7265760Z Generating XML reports...
2022-11-23T02:14:29.7266206Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021417.xml
2022-11-23T02:14:29.7266225Z 
2022-11-23T02:14:29.7266602Z ##[endgroup]
2022-11-23T02:14:29.7267065Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_c7lcjva8)
2022-11-23T02:14:29.7267086Z 
2022-11-23T02:14:29.7267196Z Shard 2: gloo should be run in 1
2022-11-23T02:14:29.7267322Z Shard 2: ucc should be run in 3
2022-11-23T02:14:29.7267597Z Running distributed/rpc/test_tensorpipe_agent ... [2022-11-23 02:14:29.516603]
2022-11-23T02:14:29.7268165Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:14:29.516950]
2022-11-23T02:14:32.1121177Z 
2022-11-23T02:14:32.1121836Z Expand the folded group to see the log file of distributed/rpc/test_tensorpipe_agent
2022-11-23T02:14:32.1122770Z ##[group]PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_5s29o0z4)
2022-11-23T02:14:32.1123392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwcdq36n
2022-11-23T02:14:32.1124325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwcdq36n/_remote_module_non_scriptable.py
2022-11-23T02:14:32.1124639Z 
2022-11-23T02:14:32.1124935Z ##[endgroup]
2022-11-23T02:14:32.1125701Z FINISHED PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_5s29o0z4)
2022-11-23T02:14:32.1126390Z 
2022-11-23T02:14:32.1126762Z Running distributed/pipeline/sync/test_transparency ... [2022-11-23 02:14:32.112164]
2022-11-23T02:14:32.1127397Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-11-23 02:14:32.112415]
2022-11-23T02:14:35.1708059Z 
2022-11-23T02:14:35.1708818Z Expand the folded group to see the log file of distributed/pipeline/sync/test_transparency
2022-11-23T02:14:35.1709909Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency__caojw2g)
2022-11-23T02:14:35.1710638Z ============================= test session starts ==============================
2022-11-23T02:14:35.1711515Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:14:35.1712109Z cachedir: .pytest_cache
2022-11-23T02:14:35.1712667Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:14:35.1713105Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:14:35.1713576Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:14:35.1714364Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:14:35.1714766Z collecting ... collected 1 item
2022-11-23T02:14:35.1715178Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears
2022-11-23T02:14:35.1715453Z 
2022-11-23T02:14:35.1715679Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2022-11-23T02:14:35.1715933Z 
2022-11-23T02:14:35.1716073Z ============================== 1 passed in 0.15s ===============================
2022-11-23T02:14:35.1716271Z 
2022-11-23T02:14:35.1716594Z ##[endgroup]
2022-11-23T02:14:35.1717247Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency__caojw2g)
2022-11-23T02:14:35.1717642Z 
2022-11-23T02:14:35.1717914Z Running distributed/pipeline/sync/test_pipe ... [2022-11-23 02:14:35.170938]
2022-11-23T02:14:35.1718511Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-11-23 02:14:35.171249]
2022-11-23T02:14:42.4938662Z 
2022-11-23T02:14:42.4939666Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipe
2022-11-23T02:14:42.4941341Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_ky92aeqp)
2022-11-23T02:14:42.4942220Z ============================= test session starts ==============================
2022-11-23T02:14:42.4943246Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:14:42.4943959Z cachedir: .pytest_cache
2022-11-23T02:14:42.4945280Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:14:42.4946109Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:14:42.4946702Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:14:42.4947745Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:14:42.4948170Z collecting ... collected 56 items
2022-11-23T02:14:42.4954426Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper
2022-11-23T02:14:42.4960031Z 
2022-11-23T02:14:42.4960247Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED     [  1%]
2022-11-23T02:14:42.4960680Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  3%]
2022-11-23T02:14:42.4961104Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
2022-11-23T02:14:42.4961541Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  7%]
2022-11-23T02:14:42.4961948Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [  8%]
2022-11-23T02:14:42.4962385Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%]
2022-11-23T02:14:42.4962826Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 12%]
2022-11-23T02:14:42.4963224Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 14%]
2022-11-23T02:14:42.4963662Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%]
2022-11-23T02:14:42.4964121Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%]
2022-11-23T02:14:42.4964567Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 19%]
2022-11-23T02:14:42.4964996Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%]
2022-11-23T02:14:42.4965493Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 23%]
2022-11-23T02:14:42.4965901Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 25%]
2022-11-23T02:14:42.4966326Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%]
2022-11-23T02:14:42.4966764Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 28%]
2022-11-23T02:14:42.4967180Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 30%]
2022-11-23T02:14:42.4967600Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%]
2022-11-23T02:14:42.4968004Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 33%]
2022-11-23T02:14:42.4968415Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 35%]
2022-11-23T02:14:42.4968831Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 37%]
2022-11-23T02:14:42.4969231Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 39%]
2022-11-23T02:14:42.4969664Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%]
2022-11-23T02:14:42.4970107Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%]
2022-11-23T02:14:42.4970562Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%]
2022-11-23T02:14:42.4970989Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%]
2022-11-23T02:14:42.4971431Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%]
2022-11-23T02:14:42.4971883Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%]
2022-11-23T02:14:42.4972319Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%]
2022-11-23T02:14:42.4972765Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%]
2022-11-23T02:14:42.4973220Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%]
2022-11-23T02:14:42.4973667Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED      [ 57%]
2022-11-23T02:14:42.4974069Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED     [ 58%]
2022-11-23T02:14:42.4974544Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%]
2022-11-23T02:14:42.4975001Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%]
2022-11-23T02:14:42.4975456Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%]
2022-11-23T02:14:42.4975902Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%]
2022-11-23T02:14:42.4976368Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%]
2022-11-23T02:14:42.4976834Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
2022-11-23T02:14:42.4977265Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 71%]
2022-11-23T02:14:42.4977679Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 73%]
2022-11-23T02:14:42.4978535Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions PASSED    [ 75%]
2022-11-23T02:14:42.4978972Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 76%]
2022-11-23T02:14:42.4979366Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 78%]
2022-11-23T02:14:42.4979778Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 80%]
2022-11-23T02:14:42.4980219Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%]
2022-11-23T02:14:42.4980668Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%]
2022-11-23T02:14:42.4981138Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device PASSED [ 85%]
2022-11-23T02:14:42.4981705Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules PASSED [ 87%]
2022-11-23T02:14:42.4982188Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%]
2022-11-23T02:14:42.4982646Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [ 91%]
2022-11-23T02:14:42.4983079Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] PASSED [ 92%]
2022-11-23T02:14:42.4983523Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] PASSED [ 94%]
2022-11-23T02:14:42.4983960Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] PASSED [ 96%]
2022-11-23T02:14:42.4984408Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device PASSED  [ 98%]
2022-11-23T02:14:42.4984835Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper PASSED  [100%]
2022-11-23T02:14:42.4985081Z 
2022-11-23T02:14:42.4985236Z ============================== 56 passed in 4.77s ==============================
2022-11-23T02:14:42.4985430Z 
2022-11-23T02:14:42.4985751Z ##[endgroup]
2022-11-23T02:14:42.4986401Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_ky92aeqp)
2022-11-23T02:14:42.4986777Z 
2022-11-23T02:14:42.4987079Z Running distributed/pipeline/sync/test_inplace ... [2022-11-23 02:14:42.494161]
2022-11-23T02:14:42.4987665Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-11-23 02:14:42.494514]
2022-11-23T02:14:45.1149048Z 
2022-11-23T02:14:45.1150248Z Expand the folded group to see the log file of distributed/pipeline/sync/test_inplace
2022-11-23T02:14:45.1151862Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_ptonxj9m)
2022-11-23T02:14:45.1152820Z ============================= test session starts ==============================
2022-11-23T02:14:45.1153831Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:14:45.1154488Z cachedir: .pytest_cache
2022-11-23T02:14:45.1155868Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:14:45.1156655Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:14:45.1157034Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:14:45.1157626Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:14:45.1158024Z collecting ... collected 3 items
2022-11-23T02:14:45.1158642Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad
2022-11-23T02:14:45.1159226Z 
2022-11-23T02:14:45.1159682Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2022-11-23T02:14:45.1160301Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2022-11-23T02:14:45.1160768Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2022-11-23T02:14:45.1161027Z 
2022-11-23T02:14:45.1161193Z ========================= 1 passed, 2 xfailed in 0.21s =========================
2022-11-23T02:14:45.1161376Z 
2022-11-23T02:14:45.1161709Z ##[endgroup]
2022-11-23T02:14:45.1162364Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_ptonxj9m)
2022-11-23T02:14:45.1162748Z 
2022-11-23T02:14:45.1163023Z Running distributed/pipeline/sync/test_copy ... [2022-11-23 02:14:45.115072]
2022-11-23T02:14:45.1163598Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-11-23 02:14:45.115427]
2022-11-23T02:14:49.9651148Z 
2022-11-23T02:14:49.9651854Z Expand the folded group to see the log file of distributed/pipeline/sync/test_copy
2022-11-23T02:14:49.9653170Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_0i7pk_tk)
2022-11-23T02:14:49.9653734Z ============================= test session starts ==============================
2022-11-23T02:14:49.9654335Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:14:49.9654687Z cachedir: .pytest_cache
2022-11-23T02:14:49.9655237Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:14:49.9655672Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:14:49.9655996Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:14:49.9656563Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:14:49.9656946Z collecting ... collected 5 items
2022-11-23T02:14:49.9657704Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors
2022-11-23T02:14:49.9658660Z 
2022-11-23T02:14:49.9658882Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
2022-11-23T02:14:49.9659320Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda PASSED   [ 40%]
2022-11-23T02:14:49.9659737Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu PASSED   [ 60%]
2022-11-23T02:14:49.9660144Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda PASSED  [ 80%]
2022-11-23T02:14:49.9660573Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]
2022-11-23T02:14:49.9660827Z 
2022-11-23T02:14:49.9660982Z ============================== 5 passed in 2.34s ===============================
2022-11-23T02:14:49.9661171Z 
2022-11-23T02:14:49.9663305Z ##[endgroup]
2022-11-23T02:14:49.9664187Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_0i7pk_tk)
2022-11-23T02:14:49.9664580Z 
2022-11-23T02:14:49.9664871Z Running distributed/pipeline/sync/test_balance ... [2022-11-23 02:14:49.965116]
2022-11-23T02:14:49.9665478Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-11-23 02:14:49.965370]
2022-11-23T02:14:59.4270773Z 
2022-11-23T02:14:59.4271514Z Expand the folded group to see the log file of distributed/pipeline/sync/test_balance
2022-11-23T02:14:59.4272597Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_cq2gwj96)
2022-11-23T02:14:59.4273148Z ============================= test session starts ==============================
2022-11-23T02:14:59.4273743Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:14:59.4274131Z cachedir: .pytest_cache
2022-11-23T02:14:59.4274679Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:14:59.4275111Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:14:59.4275434Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:14:59.4275993Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:14:59.4276379Z collecting ... collected 18 items
2022-11-23T02:14:59.4278483Z Running 18 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad
2022-11-23T02:14:59.4280776Z 
2022-11-23T02:14:59.4280983Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  5%]
2022-11-23T02:14:59.4281437Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%]
2022-11-23T02:14:59.4281924Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%]
2022-11-23T02:14:59.4282414Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%]
2022-11-23T02:14:59.4282864Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%]
2022-11-23T02:14:59.4283309Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%]
2022-11-23T02:14:59.4283782Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%]
2022-11-23T02:14:59.4284241Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%]
2022-11-23T02:14:59.4284764Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%]
2022-11-23T02:14:59.4285234Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%]
2022-11-23T02:14:59.4285688Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%]
2022-11-23T02:14:59.4286118Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%]
2022-11-23T02:14:59.4286582Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%]
2022-11-23T02:14:59.4287052Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%]
2022-11-23T02:14:59.4287502Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 83%]
2022-11-23T02:14:59.4287923Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%]
2022-11-23T02:14:59.4288365Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%]
2022-11-23T02:14:59.4288803Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2022-11-23T02:14:59.4289045Z 
2022-11-23T02:14:59.4289195Z ======================== 16 passed, 2 skipped in 6.87s =========================
2022-11-23T02:14:59.4289395Z 
2022-11-23T02:14:59.4289704Z ##[endgroup]
2022-11-23T02:14:59.4290349Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_cq2gwj96)
2022-11-23T02:14:59.4290722Z 
2022-11-23T02:14:59.4291015Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-11-23 02:14:59.427104]
2022-11-23T02:14:59.4291626Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-11-23 02:14:59.427362]
2022-11-23T02:15:01.8745938Z 
2022-11-23T02:15:01.8746419Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_stash_pop
2022-11-23T02:15:01.8747446Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_f84b7_un)
2022-11-23T02:15:01.8748029Z ============================= test session starts ==============================
2022-11-23T02:15:01.8749009Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:15:01.8749370Z cachedir: .pytest_cache
2022-11-23T02:15:01.8750143Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:15:01.8750621Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:15:01.8750938Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:15:01.8751690Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:15:01.8752091Z collecting ... collected 7 items
2022-11-23T02:15:01.8753276Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none
2022-11-23T02:15:01.8754099Z 
2022-11-23T02:15:01.8754316Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
2022-11-23T02:15:01.8754745Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
2022-11-23T02:15:01.8755178Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
2022-11-23T02:15:01.8755638Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
2022-11-23T02:15:01.8756367Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
2022-11-23T02:15:01.8756850Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
2022-11-23T02:15:01.8757278Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]
2022-11-23T02:15:01.8757522Z 
2022-11-23T02:15:01.8757697Z ============================== 7 passed in 0.06s ===============================
2022-11-23T02:15:01.8757888Z 
2022-11-23T02:15:01.8758211Z ##[endgroup]
2022-11-23T02:15:01.8758876Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_f84b7_un)
2022-11-23T02:15:01.8759281Z 
2022-11-23T02:15:01.8759604Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-11-23 02:15:01.874663]
2022-11-23T02:15:01.8760262Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-11-23 02:15:01.874921]
2022-11-23T02:15:04.2723488Z 
2022-11-23T02:15:04.2724426Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-11-23T02:15:04.2725477Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_eu9qn499)
2022-11-23T02:15:04.2726066Z ============================= test session starts ==============================
2022-11-23T02:15:04.2726858Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:15:04.2727205Z cachedir: .pytest_cache
2022-11-23T02:15:04.2727857Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:15:04.2728734Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:15:04.2729200Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:15:04.2729801Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:15:04.2730526Z collecting ... collected 6 items
2022-11-23T02:15:04.2731562Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace
2022-11-23T02:15:04.2732396Z 
2022-11-23T02:15:04.2732636Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2022-11-23T02:15:04.2733121Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2022-11-23T02:15:04.2733605Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2022-11-23T02:15:04.2734112Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2022-11-23T02:15:04.2734619Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2022-11-23T02:15:04.2735112Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2022-11-23T02:15:04.2735357Z 
2022-11-23T02:15:04.2735514Z ============================== 6 passed in 0.04s ===============================
2022-11-23T02:15:04.2735710Z 
2022-11-23T02:15:04.2736028Z ##[endgroup]
2022-11-23T02:15:04.2736752Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_eu9qn499)
2022-11-23T02:15:04.2737182Z 
2022-11-23T02:15:04.2737606Z Running distributed/optim/test_apply_optimizer_in_backward ... [2022-11-23 02:15:04.272460]
2022-11-23T02:15:04.2739005Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:04.272730]
2022-11-23T02:15:06.4012426Z 
2022-11-23T02:15:06.4013305Z Expand the folded group to see the log file of distributed/optim/test_apply_optimizer_in_backward
2022-11-23T02:15:06.4014312Z ##[group]PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_ucg0vp83)
2022-11-23T02:15:06.4014719Z 
2022-11-23T02:15:06.4015018Z ##[endgroup]
2022-11-23T02:15:06.4015987Z FINISHED PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_ucg0vp83)
2022-11-23T02:15:06.4016550Z 
2022-11-23T02:15:06.4016915Z Running distributed/elastic/events/lib_test ... [2022-11-23 02:15:06.401307]
2022-11-23T02:15:06.4018977Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-11-23 02:15:06.401578]
2022-11-23T02:15:08.9516835Z 
2022-11-23T02:15:08.9517696Z Expand the folded group to see the log file of distributed/elastic/events/lib_test
2022-11-23T02:15:08.9518654Z ##[group]PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_i3msmnkp)
2022-11-23T02:15:08.9519198Z ============================= test session starts ==============================
2022-11-23T02:15:08.9519872Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:15:08.9520622Z cachedir: .pytest_cache
2022-11-23T02:15:08.9521211Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:15:08.9521753Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:15:08.9522349Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:15:08.9522926Z collecting ... collected 8 items
2022-11-23T02:15:08.9524162Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str
2022-11-23T02:15:08.9525255Z 
2022-11-23T02:15:08.9525481Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%]
2022-11-23T02:15:08.9525920Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%]
2022-11-23T02:15:08.9526377Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%]
2022-11-23T02:15:08.9526874Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%]
2022-11-23T02:15:08.9527411Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%]
2022-11-23T02:15:08.9527947Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%]
2022-11-23T02:15:08.9528430Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%]
2022-11-23T02:15:08.9529026Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%]
2022-11-23T02:15:08.9529307Z 
2022-11-23T02:15:08.9529446Z ============================== 8 passed in 1.81s ===============================
2022-11-23T02:15:08.9529641Z 
2022-11-23T02:15:08.9529955Z ##[endgroup]
2022-11-23T02:15:08.9530583Z FINISHED PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_i3msmnkp)
2022-11-23T02:15:08.9530946Z 
2022-11-23T02:15:08.9531221Z Running distributed/_shard/test_sharder ... [2022-11-23 02:15:08.951854]
2022-11-23T02:15:08.9531870Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:08.952144]
2022-11-23T02:15:11.0521087Z 
2022-11-23T02:15:11.0521575Z Expand the folded group to see the log file of distributed/_shard/test_sharder
2022-11-23T02:15:11.0522508Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_esgtp0bz)
2022-11-23T02:15:11.0523110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpld6ebhje
2022-11-23T02:15:11.0523656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpld6ebhje/_remote_module_non_scriptable.py
2022-11-23T02:15:11.0523960Z 
2022-11-23T02:15:11.0524244Z ##[endgroup]
2022-11-23T02:15:11.0524948Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_esgtp0bz)
2022-11-23T02:15:11.0525289Z 
2022-11-23T02:15:11.0525551Z Running distributed/test_launcher ... [2022-11-23 02:15:11.052183]
2022-11-23T02:15:11.0527819Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:11.052462]
2022-11-23T02:15:15.4960126Z 
2022-11-23T02:15:15.4960648Z Expand the folded group to see the log file of distributed/test_launcher
2022-11-23T02:15:15.4961596Z ##[group]PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_91kacdbw)
2022-11-23T02:15:15.4962117Z 
2022-11-23T02:15:15.4962287Z Running tests...
2022-11-23T02:15:15.4963199Z ----------------------------------------------------------------------
2022-11-23T02:15:15.4964212Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher
2022-11-23T02:15:15.4966446Z   test_launch_user_script (__main__.TestDistributedLaunch) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79488 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.667s)
2022-11-23T02:15:15.4967596Z 
2022-11-23T02:15:15.4968107Z ----------------------------------------------------------------------
2022-11-23T02:15:15.4968698Z Ran 1 test in 1.667s
2022-11-23T02:15:15.4969014Z 
2022-11-23T02:15:15.4969211Z OK (skipped=1)
2022-11-23T02:15:15.4969502Z 
2022-11-23T02:15:15.4969732Z Generating XML reports...
2022-11-23T02:15:15.4970895Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20221123021513.xml
2022-11-23T02:15:15.4971567Z 
2022-11-23T02:15:15.4972157Z ##[endgroup]
2022-11-23T02:15:15.4973236Z FINISHED PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_91kacdbw)
2022-11-23T02:15:15.4973891Z 
2022-11-23T02:15:15.4974454Z Running distributed/checkpoint/test_planner ... [2022-11-23 02:15:15.496167]
2022-11-23T02:15:15.4975744Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:15.496516]
2022-11-23T02:15:19.4717570Z 
2022-11-23T02:15:19.4718241Z Expand the folded group to see the log file of distributed/checkpoint/test_planner
2022-11-23T02:15:19.4719658Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_mz4qicfq)
2022-11-23T02:15:19.4720098Z 
2022-11-23T02:15:19.4720259Z Running tests...
2022-11-23T02:15:19.4720897Z ----------------------------------------------------------------------
2022-11-23T02:15:19.4721591Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_planner
2022-11-23T02:15:19.4722183Z   test_global_plan (__main__.TestSavePlan) ... ok (1.660s)
2022-11-23T02:15:19.4722558Z   test_load_with_resharding (__main__.TestSavePlan) ... ok (0.004s)
2022-11-23T02:15:19.4722951Z   test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s)
2022-11-23T02:15:19.4723363Z   test_local_load_plan (__main__.TestSavePlan) ... ok (0.004s)
2022-11-23T02:15:19.4723719Z   test_local_plan (__main__.TestSavePlan) ... ok (0.003s)
2022-11-23T02:15:19.4723922Z 
2022-11-23T02:15:19.4724185Z ----------------------------------------------------------------------
2022-11-23T02:15:19.4724516Z Ran 5 tests in 1.675s
2022-11-23T02:15:19.4724677Z 
2022-11-23T02:15:19.4724770Z OK
2022-11-23T02:15:19.4724900Z 
2022-11-23T02:15:19.4725023Z Generating XML reports...
2022-11-23T02:15:19.4725603Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20221123021517.xml
2022-11-23T02:15:19.4725947Z 
2022-11-23T02:15:19.4726271Z ##[endgroup]
2022-11-23T02:15:19.4726879Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_mz4qicfq)
2022-11-23T02:15:19.4727396Z 
2022-11-23T02:15:19.4727663Z Running distributed/_composable/test_contract ... [2022-11-23 02:15:19.471768]
2022-11-23T02:15:19.4728349Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_contract.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:19.472033]
2022-11-23T02:15:23.4852457Z 
2022-11-23T02:15:23.4852971Z Expand the folded group to see the log file of distributed/_composable/test_contract
2022-11-23T02:15:23.4854023Z ##[group]PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_xbd9h6ei)
2022-11-23T02:15:23.4854527Z 
2022-11-23T02:15:23.4854630Z Running tests...
2022-11-23T02:15:23.4855148Z ----------------------------------------------------------------------
2022-11-23T02:15:23.4855714Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_contract
2022-11-23T02:15:23.4856164Z   test_add_hooks (__main__.TestContract) ... ok (1.688s)
2022-11-23T02:15:23.4856502Z   test_modify_fqn (__main__.TestContract) ... ok (0.002s)
2022-11-23T02:15:23.4856840Z   test_state (__main__.TestContract) ... ok (0.003s)
2022-11-23T02:15:23.4857037Z 
2022-11-23T02:15:23.4857305Z ----------------------------------------------------------------------
2022-11-23T02:15:23.4857618Z Ran 3 tests in 1.693s
2022-11-23T02:15:23.4857781Z 
2022-11-23T02:15:23.4857876Z OK
2022-11-23T02:15:23.4858360Z 
2022-11-23T02:15:23.4858491Z Generating XML reports...
2022-11-23T02:15:23.4859096Z Generated XML report: test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20221123021521.xml
2022-11-23T02:15:23.4859416Z 
2022-11-23T02:15:23.4859737Z ##[endgroup]
2022-11-23T02:15:23.4860346Z FINISHED PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_xbd9h6ei)
2022-11-23T02:15:23.4860704Z 
2022-11-23T02:15:23.4860975Z Running distributed/test_multi_threaded_pg ... [2022-11-23 02:15:23.485263]
2022-11-23T02:15:23.4861646Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:23.485563]
2022-11-23T02:15:27.4874610Z 
2022-11-23T02:15:27.4875482Z Expand the folded group to see the log file of distributed/test_multi_threaded_pg
2022-11-23T02:15:27.4876517Z ##[group]PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_multi_threaded_pg_50j4nez3)
2022-11-23T02:15:27.4876900Z 
2022-11-23T02:15:27.4877017Z Running tests...
2022-11-23T02:15:27.4877551Z ----------------------------------------------------------------------
2022-11-23T02:15:27.4878117Z Test results will be stored in test-reports/python-unittest/distributed.test_multi_threaded_pg
2022-11-23T02:15:27.4878657Z   test_allgather (__main__.TestCollectivesWithBaseClass) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:15:27.4879204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:27.4879721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:27.4880226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:15:27.4880739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:15:27.4881455Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4882184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4882887Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4883606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4884149Z ok (1.654s)
2022-11-23T02:15:27.4884621Z   test_broadcast (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:27.4885232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:27.4885750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:15:27.4886265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:15:27.4886936Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4887688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4888411Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4889126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4889539Z ok (0.020s)
2022-11-23T02:15:27.4890018Z   test_broadcast_object_list (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:27.4890634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:27.4891149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:15:27.4891643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:15:27.4892372Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4893099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4893881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4894594Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4895034Z 3 -> 4
2022-11-23T02:15:27.4895288Z 0 -> 4
2022-11-23T02:15:27.4895515Z 1 -> 4
2022-11-23T02:15:27.4895760Z 2 -> 4
2022-11-23T02:15:27.4895982Z ok (0.016s)
2022-11-23T02:15:27.4896456Z   test_reduce_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:27.4897067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:27.4897587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:15:27.4898388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:15:27.4899074Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4899791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4900508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4901225Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4901620Z ok (0.015s)
2022-11-23T02:15:27.4902092Z   test_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:27.4902788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:27.4903302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:15:27.4903801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:15:27.4904488Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4905207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4905926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4906628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4907043Z ok (0.014s)
2022-11-23T02:15:27.4907542Z   test_broadcast_object_list (__main__.TestCollectivesWithWrapper) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:27.4908151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:27.4908663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:15:27.4909174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:15:27.4909853Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4910559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4911280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4912003Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:15:27.4912412Z ok (0.016s)
2022-11-23T02:15:27.4912615Z 
2022-11-23T02:15:27.4912911Z ----------------------------------------------------------------------
2022-11-23T02:15:27.4913253Z Ran 6 tests in 1.736s
2022-11-23T02:15:27.4913424Z 
2022-11-23T02:15:27.4913519Z OK
2022-11-23T02:15:27.4913657Z 
2022-11-23T02:15:27.4913767Z Generating XML reports...
2022-11-23T02:15:27.4914430Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20221123021525.xml
2022-11-23T02:15:27.4915303Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20221123021525.xml
2022-11-23T02:15:27.4915698Z 
2022-11-23T02:15:27.4916009Z ##[endgroup]
2022-11-23T02:15:27.4916637Z FINISHED PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_multi_threaded_pg_50j4nez3)
2022-11-23T02:15:27.4917003Z 
2022-11-23T02:15:27.4917291Z Running distributed/elastic/utils/util_test ... [2022-11-23 02:15:27.487499]
2022-11-23T02:15:27.4918003Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:27.487792]
2022-11-23T02:15:31.5594372Z 
2022-11-23T02:15:31.5594983Z Expand the folded group to see the log file of distributed/elastic/utils/util_test
2022-11-23T02:15:31.5595912Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_1h4guytv)
2022-11-23T02:15:31.5596312Z 
2022-11-23T02:15:31.5596426Z Running tests...
2022-11-23T02:15:31.5596948Z ----------------------------------------------------------------------
2022-11-23T02:15:31.5597805Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test
2022-11-23T02:15:31.5598234Z   test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (1.659s)
2022-11-23T02:15:31.5598589Z   test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s)
2022-11-23T02:15:31.5598957Z   test_synchronize (__main__.StoreUtilTest) ... ok (0.003s)
2022-11-23T02:15:31.5599275Z   test_get_logger (__main__.UtilTest) ... ok (0.108s)
2022-11-23T02:15:31.5599621Z   test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s)
2022-11-23T02:15:31.5599980Z   test_get_logger_different (__main__.UtilTest) ... ok (0.001s)
2022-11-23T02:15:31.5600329Z   test_get_logger_none (__main__.UtilTest) ... ok (0.001s)
2022-11-23T02:15:31.5600514Z 
2022-11-23T02:15:31.5600785Z ----------------------------------------------------------------------
2022-11-23T02:15:31.5601111Z Ran 7 tests in 1.777s
2022-11-23T02:15:31.5601272Z 
2022-11-23T02:15:31.5601366Z OK
2022-11-23T02:15:31.5601504Z 
2022-11-23T02:15:31.5601625Z Generating XML reports...
2022-11-23T02:15:31.5602227Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221123021529.xml
2022-11-23T02:15:31.5602962Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221123021529.xml
2022-11-23T02:15:31.5603289Z 
2022-11-23T02:15:31.5603601Z ##[endgroup]
2022-11-23T02:15:31.5604189Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_1h4guytv)
2022-11-23T02:15:31.5604550Z 
2022-11-23T02:15:31.5604811Z Running distributed/rpc/test_share_memory ... [2022-11-23 02:15:31.559434]
2022-11-23T02:15:31.5605510Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:31.559710]
2022-11-23T02:15:40.3887256Z 
2022-11-23T02:15:40.3887800Z Expand the folded group to see the log file of distributed/rpc/test_share_memory
2022-11-23T02:15:40.3888894Z ##[group]PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_4vhslr2c)
2022-11-23T02:15:40.3889648Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:15:40.3890054Z <unittest.suite.TestSuite tests=[<__main__.TestRPCPickler testMethod=test_case>]>
2022-11-23T02:15:40.3890427Z test_case (__main__.TestRPCPickler)
2022-11-23T02:15:40.3891082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:40.3891513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:40.3892129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:40.3892595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:40.3892829Z 
2022-11-23T02:15:40.3892942Z Running tests...
2022-11-23T02:15:40.3893328Z ----------------------------------------------------------------------
2022-11-23T02:15:40.3893870Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory
2022-11-23T02:15:40.3894582Z   test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:40.3895089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:40.3895639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:40.3896104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:40.3896944Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:15:40.3897721Z   warnings.warn(message, UserWarning)
2022-11-23T02:15:40.3898650Z ok (4.425s)
2022-11-23T02:15:40.3898812Z 
2022-11-23T02:15:40.3899093Z ----------------------------------------------------------------------
2022-11-23T02:15:40.3899419Z Ran 1 test in 4.425s
2022-11-23T02:15:40.3899579Z 
2022-11-23T02:15:40.3899673Z OK
2022-11-23T02:15:40.3899789Z 
2022-11-23T02:15:40.3899914Z Generating XML reports...
2022-11-23T02:15:40.3900494Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20221123021535.xml
2022-11-23T02:15:40.3900834Z 
2022-11-23T02:15:40.3901153Z ##[endgroup]
2022-11-23T02:15:40.3901724Z FINISHED PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_4vhslr2c)
2022-11-23T02:15:40.3902070Z 
2022-11-23T02:15:40.3902347Z Running distributed/fsdp/test_fsdp_traversal ... [2022-11-23 02:15:40.388761]
2022-11-23T02:15:40.3903031Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:40.389047]
2022-11-23T02:15:48.2550345Z 
2022-11-23T02:15:48.2551462Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal
2022-11-23T02:15:48.2553061Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_90o0fqzk)
2022-11-23T02:15:48.2553505Z 
2022-11-23T02:15:48.2553662Z Running tests...
2022-11-23T02:15:48.2554291Z ----------------------------------------------------------------------
2022-11-23T02:15:48.2568862Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2022-11-23T02:15:48.2569781Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:15:48.2570801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43177
2022-11-23T02:15:48.2571408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43178
2022-11-23T02:15:48.2572302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:48.2572789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:48.2573401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:48.2573902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:48.2574511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:48.2574963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:48.2575563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:48.2576063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:48.2576519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:48.2577046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:48.2577734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:48.2578938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:48.2579446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:48.2579912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:48.2580265Z dist init r=0, world=2
2022-11-23T02:15:48.2580515Z dist init r=1, world=2
2022-11-23T02:15:48.2580868Z ok (5.529s)
2022-11-23T02:15:48.2581016Z 
2022-11-23T02:15:48.2581291Z ----------------------------------------------------------------------
2022-11-23T02:15:48.2581618Z Ran 1 test in 5.529s
2022-11-23T02:15:48.2581760Z 
2022-11-23T02:15:48.2581855Z OK
2022-11-23T02:15:48.2581991Z 
2022-11-23T02:15:48.2582117Z Generating XML reports...
2022-11-23T02:15:48.2582704Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123021542.xml
2022-11-23T02:15:48.2583047Z 
2022-11-23T02:15:48.2583369Z ##[endgroup]
2022-11-23T02:15:48.2583997Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_90o0fqzk)
2022-11-23T02:15:48.2584359Z 
2022-11-23T02:15:48.2584617Z Running distributed/fsdp/test_fsdp_fx ... [2022-11-23 02:15:48.255093]
2022-11-23T02:15:48.2585273Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:48.255366]
2022-11-23T02:15:56.1551758Z 
2022-11-23T02:15:56.1552489Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx
2022-11-23T02:15:56.1554126Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_7zf89f2d)
2022-11-23T02:15:56.1554944Z 
2022-11-23T02:15:56.1555119Z Running tests...
2022-11-23T02:15:56.1555711Z ----------------------------------------------------------------------
2022-11-23T02:15:56.1556264Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx
2022-11-23T02:15:56.1556710Z   test_symbolic_tracing_outputs (__main__.TestSymbolicTracing)
2022-11-23T02:15:56.1557189Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:15:56.1557705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43291
2022-11-23T02:15:56.1558159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43292
2022-11-23T02:15:56.1558761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:56.1559451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:56.1560053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:56.1560516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:56.1561073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:56.1561516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:56.1562084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:56.1562529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:56.1562970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:56.1563485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:56.1564140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:56.1564823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:56.1565326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:56.1565793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:56.1566148Z dist init r=0, world=2
2022-11-23T02:15:56.1566382Z dist init r=1, world=2
2022-11-23T02:15:56.1566727Z ok (5.574s)
2022-11-23T02:15:56.1566876Z 
2022-11-23T02:15:56.1567149Z ----------------------------------------------------------------------
2022-11-23T02:15:56.1567456Z Ran 1 test in 5.574s
2022-11-23T02:15:56.1567616Z 
2022-11-23T02:15:56.1567710Z OK
2022-11-23T02:15:56.1567844Z 
2022-11-23T02:15:56.1567973Z Generating XML reports...
2022-11-23T02:15:56.1568568Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123021550.xml
2022-11-23T02:15:56.1568901Z 
2022-11-23T02:15:56.1569225Z ##[endgroup]
2022-11-23T02:15:56.1569799Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_7zf89f2d)
2022-11-23T02:15:56.1570140Z 
2022-11-23T02:15:56.1570428Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-11-23 02:15:56.155134]
2022-11-23T02:15:56.1571117Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:56.155427]
2022-11-23T02:16:04.5575274Z 
2022-11-23T02:16:04.5576331Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T02:16:04.5578522Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_ceo4f3aj)
2022-11-23T02:16:04.5579154Z 
2022-11-23T02:16:04.5579274Z Running tests...
2022-11-23T02:16:04.5579810Z ----------------------------------------------------------------------
2022-11-23T02:16:04.5580391Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2022-11-23T02:16:04.5580895Z   test_multi_forward (__main__.TestMultiForward) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:04.5581358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43405
2022-11-23T02:16:04.5581781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43406
2022-11-23T02:16:04.5582401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:04.5582854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:04.5583661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:04.5584150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:04.5584731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:04.5585173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:04.5585722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:04.5586182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:04.5586637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:04.5587126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:04.5587768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:04.5588447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:04.5588964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:04.5589411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:04.5589882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:04.5590364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:04.5591749Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:16:04.5592577Z   warnings.warn(
2022-11-23T02:16:04.5593712Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:16:04.5594489Z   warnings.warn(
2022-11-23T02:16:04.5594735Z dist init r=1, world=2
2022-11-23T02:16:04.5594985Z dist init r=0, world=2
2022-11-23T02:16:04.5595202Z ok (6.071s)
2022-11-23T02:16:04.5595349Z 
2022-11-23T02:16:04.5595622Z ----------------------------------------------------------------------
2022-11-23T02:16:04.5595948Z Ran 1 test in 6.071s
2022-11-23T02:16:04.5596107Z 
2022-11-23T02:16:04.5596183Z OK
2022-11-23T02:16:04.5596314Z 
2022-11-23T02:16:04.5596437Z Generating XML reports...
2022-11-23T02:16:04.5597040Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123021558.xml
2022-11-23T02:16:04.5597399Z 
2022-11-23T02:16:04.5597731Z ##[endgroup]
2022-11-23T02:16:04.5598347Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_ceo4f3aj)
2022-11-23T02:16:04.5598729Z 
2022-11-23T02:16:04.5599034Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-11-23 02:16:04.557585]
2022-11-23T02:16:04.5599842Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:16:04.557898]
2022-11-23T02:16:13.3470749Z 
2022-11-23T02:16:13.3471270Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T02:16:13.3472553Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bh698ist)
2022-11-23T02:16:13.3473332Z 
2022-11-23T02:16:13.3473548Z Running tests...
2022-11-23T02:16:13.3474195Z ----------------------------------------------------------------------
2022-11-23T02:16:13.3474803Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding
2022-11-23T02:16:13.3475371Z   test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:13.3475871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43523
2022-11-23T02:16:13.3476302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43524
2022-11-23T02:16:13.3476735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43525
2022-11-23T02:16:13.3477179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43526
2022-11-23T02:16:13.3477783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:13.3478234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:13.3478807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:13.3479282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:13.3480149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:13.3480599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:13.3481170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:13.3481634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:13.3482188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:13.3482627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:13.3483255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:13.3483714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:13.3484273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:13.3484712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:13.3485283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:13.3485722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:13.3486159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:13.3486631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:13.3487096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:13.3487538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:13.3487917Z skip: Need at least 4 CUDA devices (4.059s)
2022-11-23T02:16:13.3488408Z   test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43659
2022-11-23T02:16:13.3488922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43660
2022-11-23T02:16:13.3489461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43661
2022-11-23T02:16:13.3489909Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43662
2022-11-23T02:16:13.3490516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:13.3490943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:13.3491512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:13.3492026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:13.3492589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:13.3493031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:13.3493600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:13.3494063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:13.3494619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:13.3495057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:13.3495617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:13.3496069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:13.3496614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:13.3497122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:13.3497691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:13.3498391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:13.3498828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:13.3499302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:13.3499766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:13.3500205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:13.3500583Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:16:13.3500782Z 
2022-11-23T02:16:13.3501060Z ----------------------------------------------------------------------
2022-11-23T02:16:13.3501369Z Ran 2 tests in 6.471s
2022-11-23T02:16:13.3501543Z 
2022-11-23T02:16:13.3501654Z OK (skipped=2)
2022-11-23T02:16:13.3501809Z 
2022-11-23T02:16:13.3501932Z Generating XML reports...
2022-11-23T02:16:13.3502577Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221123021606.xml
2022-11-23T02:16:13.3502956Z 
2022-11-23T02:16:13.3503262Z ##[endgroup]
2022-11-23T02:16:13.3503942Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bh698ist)
2022-11-23T02:16:13.3504342Z 
2022-11-23T02:16:13.3504657Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-11-23 02:16:13.347277]
2022-11-23T02:16:13.3505380Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:16:13.347651]
2022-11-23T02:16:22.1419300Z 
2022-11-23T02:16:22.1420175Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T02:16:22.1422517Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_1ypv8ntq)
2022-11-23T02:16:22.1423436Z 
2022-11-23T02:16:22.1423676Z Running tests...
2022-11-23T02:16:22.1424383Z ----------------------------------------------------------------------
2022-11-23T02:16:22.1425000Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag
2022-11-23T02:16:22.1425569Z   test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:22.1426081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43830
2022-11-23T02:16:22.1426515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43831
2022-11-23T02:16:22.1426952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43832
2022-11-23T02:16:22.1427396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43833
2022-11-23T02:16:22.1428005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:22.1428452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:22.1429028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:22.1429495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:22.1430057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:22.1430640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:22.1431207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:22.1431673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:22.1432224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:22.1432662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:22.1433230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:22.1433672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:22.1434242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:22.1434681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:22.1435248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:22.1435734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:22.1436153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:22.1436624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:22.1437093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:22.1437534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:22.1437920Z skip: Need at least 4 CUDA devices (4.071s)
2022-11-23T02:16:22.1438421Z   test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43966
2022-11-23T02:16:22.1438973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43967
2022-11-23T02:16:22.1439396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43968
2022-11-23T02:16:22.1439888Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43969
2022-11-23T02:16:22.1440504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:22.1440929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:22.1441496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:22.1441959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:22.1442526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:22.1442955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:22.1443519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:22.1443977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:22.1444533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:22.1444970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:22.1445532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:22.1445992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:22.1446540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:22.1446980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:22.1447965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:22.1448430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:22.1448846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:22.1449317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:22.1449779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:22.1450218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:22.1450601Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:16:22.1450794Z 
2022-11-23T02:16:22.1451070Z ----------------------------------------------------------------------
2022-11-23T02:16:22.1451399Z Ran 2 tests in 6.481s
2022-11-23T02:16:22.1451547Z 
2022-11-23T02:16:22.1451658Z OK (skipped=2)
2022-11-23T02:16:22.1451813Z 
2022-11-23T02:16:22.1451939Z Generating XML reports...
2022-11-23T02:16:22.1452601Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20221123021615.xml
2022-11-23T02:16:22.1452999Z 
2022-11-23T02:16:22.1453314Z ##[endgroup]
2022-11-23T02:16:22.1453999Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_1ypv8ntq)
2022-11-23T02:16:22.1454404Z 
2022-11-23T02:16:22.1454686Z Running distributed/_composable/test_replicate ... [2022-11-23 02:16:22.142165]
2022-11-23T02:16:22.1455384Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:16:22.142516]
2022-11-23T02:16:31.1327247Z 
2022-11-23T02:16:31.1328002Z Expand the folded group to see the log file of distributed/_composable/test_replicate
2022-11-23T02:16:31.1329886Z ##[group]PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_sl7gbg2k)
2022-11-23T02:16:31.1330611Z 
2022-11-23T02:16:31.1331127Z Running tests...
2022-11-23T02:16:31.1331885Z ----------------------------------------------------------------------
2022-11-23T02:16:31.1332801Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_replicate
2022-11-23T02:16:31.1333307Z   test_replicate_multi_module (__main__.ReplicateTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:31.1333775Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44137
2022-11-23T02:16:31.1334199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44138
2022-11-23T02:16:31.1334634Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44139
2022-11-23T02:16:31.1335085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44140
2022-11-23T02:16:31.1335686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:31.1336143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:31.1336714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:31.1337180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:31.1337733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:31.1338387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:31.1338961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:31.1339534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:31.1340113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:31.1340553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:31.1341151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:31.1341595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:31.1342168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:31.1342605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:31.1343171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:31.1343610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:31.1344047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:31.1344516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:31.1344964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:31.1345412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:31.1345894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:31.1346380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:31.1346844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:16:31.1347325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:16:31.1347978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:16:31.1348697Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:16:31.1349432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:16:31.1350122Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:16:31.1350641Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:31.1351108Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:31.1351546Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:31.1352016Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:31.1352361Z ok (4.153s)
2022-11-23T02:16:31.1352765Z   test_replicate_single_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44293
2022-11-23T02:16:31.1353288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44294
2022-11-23T02:16:31.1353729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44295
2022-11-23T02:16:31.1354168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44296
2022-11-23T02:16:31.1354756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:31.1355201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:31.1355765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:31.1356293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:31.1356853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:31.1357292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:31.1357864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:31.1358308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:31.1358883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:31.1359322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:31.1359887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:31.1360327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:31.1360901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:31.1361336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:31.1361886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:31.1362347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:31.1362781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:31.1363249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:31.1363690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:31.1364143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:31.1364625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:31.1365118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:31.1365634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:16:31.1366118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:16:31.1366771Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:16:31.1367435Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:16:31.1368114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:16:31.1368792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:16:31.1369314Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:31.1369769Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:31.1370230Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:31.1370698Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:31.1371036Z ok (2.510s)
2022-11-23T02:16:31.1371166Z 
2022-11-23T02:16:31.1371437Z ----------------------------------------------------------------------
2022-11-23T02:16:31.1371768Z Ran 2 tests in 6.663s
2022-11-23T02:16:31.1371930Z 
2022-11-23T02:16:31.1372025Z OK
2022-11-23T02:16:31.1372160Z 
2022-11-23T02:16:31.1372266Z Generating XML reports...
2022-11-23T02:16:31.1372854Z Generated XML report: test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20221123021624.xml
2022-11-23T02:16:31.1373253Z 
2022-11-23T02:16:31.1373579Z ##[endgroup]
2022-11-23T02:16:31.1374182Z FINISHED PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_sl7gbg2k)
2022-11-23T02:16:31.1374547Z 
2022-11-23T02:16:31.1374841Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-11-23 02:16:31.132872]
2022-11-23T02:16:31.1375547Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:16:31.133271]
2022-11-23T02:16:42.2945731Z 
2022-11-23T02:16:42.2946203Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_init
2022-11-23T02:16:42.2947252Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_529oh6pr)
2022-11-23T02:16:42.2947665Z 
2022-11-23T02:16:42.2947783Z Running tests...
2022-11-23T02:16:42.2948313Z ----------------------------------------------------------------------
2022-11-23T02:16:42.2948890Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init
2022-11-23T02:16:42.2949404Z   test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit)
2022-11-23T02:16:42.2949910Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:42.2950408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44484
2022-11-23T02:16:42.2950899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44485
2022-11-23T02:16:42.2951315Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44486
2022-11-23T02:16:42.2951749Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44487
2022-11-23T02:16:42.2952393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2952824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2953653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2954143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2954725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2955148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2955720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2956182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2956746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2957189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2957756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2958220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2958772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2959215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2959784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2960244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2960656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:42.2961236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:42.2961699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:42.2962142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:42.2962531Z skip: Need at least 4 CUDA devices (4.072s)
2022-11-23T02:16:42.2962910Z   test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit)
2022-11-23T02:16:42.2963427Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44620
2022-11-23T02:16:42.2963931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44621
2022-11-23T02:16:42.2964372Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44622
2022-11-23T02:16:42.2964808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44623
2022-11-23T02:16:42.2965405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2965854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2966427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2966892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2967448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2967892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2968463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2968928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2969490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2969933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2970558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2971010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2971583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2972024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2972594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2973034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2973470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:42.2973944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:42.2974390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:42.2974858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:42.2975244Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:16:42.2975622Z   test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit)
2022-11-23T02:16:42.2976119Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44756
2022-11-23T02:16:42.2976630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44757
2022-11-23T02:16:42.2977067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44758
2022-11-23T02:16:42.2977553Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44759
2022-11-23T02:16:42.2978441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2978899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2979478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2979926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2980500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2980938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2981506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2981949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2982527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2982967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2983516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2983981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2984554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:42.2984996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:42.2985544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:42.2986005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:42.2986444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:42.2986892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:42.2987440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:42.2987909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:42.2988302Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:16:42.2988476Z 
2022-11-23T02:16:42.2988754Z ----------------------------------------------------------------------
2022-11-23T02:16:42.2989086Z Ran 3 tests in 8.893s
2022-11-23T02:16:42.2989250Z 
2022-11-23T02:16:42.2989362Z OK (skipped=3)
2022-11-23T02:16:42.2989518Z 
2022-11-23T02:16:42.2989624Z Generating XML reports...
2022-11-23T02:16:42.2990269Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221123021632.xml
2022-11-23T02:16:42.2990654Z 
2022-11-23T02:16:42.2990978Z ##[endgroup]
2022-11-23T02:16:42.2991603Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_529oh6pr)
2022-11-23T02:16:42.2992028Z 
2022-11-23T02:16:42.2992324Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-11-23 02:16:42.294647]
2022-11-23T02:16:42.2993052Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:16:42.294912]
2022-11-23T02:16:54.3846708Z 
2022-11-23T02:16:54.3847438Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint
2022-11-23T02:16:54.3848430Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_hw4wcfq7)
2022-11-23T02:16:54.3849482Z 
2022-11-23T02:16:54.3849682Z Running tests...
2022-11-23T02:16:54.3850522Z ----------------------------------------------------------------------
2022-11-23T02:16:54.3851728Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2022-11-23T02:16:54.3852346Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:54.3852901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44927
2022-11-23T02:16:54.3853354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44928
2022-11-23T02:16:54.3854462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:54.3855408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:54.3856224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:54.3856894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:54.3858446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:54.3859188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:54.3859772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:54.3860242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:54.3860673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:54.3861162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:54.3861811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:54.3862484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:54.3863175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:54.3863660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:54.3864017Z dist init r=1, world=2
2022-11-23T02:16:54.3864248Z dist init r=0, world=2
2022-11-23T02:16:54.3864487Z ok (5.654s)
2022-11-23T02:16:54.3865004Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45010
2022-11-23T02:16:54.3865605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45011
2022-11-23T02:16:54.3866195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:54.3866644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:54.3867212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:54.3867680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:54.3868232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:54.3868670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:54.3869232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:54.3869672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:54.3870116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:54.3870695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:54.3871347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:54.3872017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:54.3872534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:54.3872999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:54.3873333Z dist init r=0, world=2
2022-11-23T02:16:54.3873585Z dist init r=1, world=2
2022-11-23T02:16:54.3873827Z ok (4.112s)
2022-11-23T02:16:54.3873978Z 
2022-11-23T02:16:54.3874247Z ----------------------------------------------------------------------
2022-11-23T02:16:54.3874557Z Ran 2 tests in 9.766s
2022-11-23T02:16:54.3874722Z 
2022-11-23T02:16:54.3874817Z OK
2022-11-23T02:16:54.3874950Z 
2022-11-23T02:16:54.3875075Z Generating XML reports...
2022-11-23T02:16:54.3875712Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123021644.xml
2022-11-23T02:16:54.3876106Z 
2022-11-23T02:16:54.3876446Z ##[endgroup]
2022-11-23T02:16:54.3877095Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_hw4wcfq7)
2022-11-23T02:16:54.3877482Z 
2022-11-23T02:16:54.3877775Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-11-23 02:16:54.384738]
2022-11-23T02:16:54.3878505Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:16:54.385029]
2022-11-23T02:17:07.7994419Z 
2022-11-23T02:17:07.7995456Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T02:17:07.7997189Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_ykdllzxz)
2022-11-23T02:17:07.7998250Z 
2022-11-23T02:17:07.7998494Z Running tests...
2022-11-23T02:17:07.7999357Z ----------------------------------------------------------------------
2022-11-23T02:17:07.8000468Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2022-11-23T02:17:07.8001354Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2022-11-23T02:17:07.8002155Z Test torch.allclose(ShardedTensor, ShardedTensor) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:17:07.8002818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45128
2022-11-23T02:17:07.8003249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45129
2022-11-23T02:17:07.8003697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45130
2022-11-23T02:17:07.8004135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45131
2022-11-23T02:17:07.8004785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8005531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8006256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8006730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8007285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8007725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8008294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8008933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8009499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8009945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8010511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8010973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8011528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8011965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8012531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8012976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8013410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:07.8013885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:07.8014353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:17:07.8014796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:17:07.8015177Z skip: Need at least 4 CUDA devices (4.025s)
2022-11-23T02:17:07.8015677Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45264
2022-11-23T02:17:07.8016205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45265
2022-11-23T02:17:07.8016706Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45266
2022-11-23T02:17:07.8017145Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45267
2022-11-23T02:17:07.8017809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8018759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8019335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8019801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8020380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8020818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8021368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8021833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8022403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8022825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8023391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8023847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8024412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8024829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8025394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8025953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8026365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:07.8026838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:17:07.8027293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:17:07.8027750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:07.8028120Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:17:07.8028480Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2022-11-23T02:17:07.8028979Z Test torch.equal(ShardedTensor, ShardedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45400
2022-11-23T02:17:07.8029473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45401
2022-11-23T02:17:07.8029920Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45402
2022-11-23T02:17:07.8030349Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45403
2022-11-23T02:17:07.8030959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8031389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8031948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8032392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8032942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8033402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8033982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8034444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8035000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8035527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8036102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8036560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8037116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8037559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8038121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8038564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8038996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:07.8039470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:07.8039931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:17:07.8040372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:17:07.8040754Z skip: Need at least 4 CUDA devices (2.309s)
2022-11-23T02:17:07.8041246Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45536
2022-11-23T02:17:07.8041771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45537
2022-11-23T02:17:07.8042209Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45538
2022-11-23T02:17:07.8042709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45539
2022-11-23T02:17:07.8043307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8043735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8044300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8044764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8045339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8045757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8046320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8046781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8047330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8047769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8048330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8048783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8049332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:07.8049768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:07.8050332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:07.8050769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:07.8051206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:17:07.8051671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:07.8052191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:17:07.8052640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:07.8053023Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:17:07.8053216Z 
2022-11-23T02:17:07.8053489Z ----------------------------------------------------------------------
2022-11-23T02:17:07.8053800Z Ran 4 tests in 11.153s
2022-11-23T02:17:07.8053964Z 
2022-11-23T02:17:07.8054074Z OK (skipped=4)
2022-11-23T02:17:07.8054227Z 
2022-11-23T02:17:07.8054352Z Generating XML reports...
2022-11-23T02:17:07.8055020Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221123021656.xml
2022-11-23T02:17:07.8055424Z 
2022-11-23T02:17:07.8055750Z ##[endgroup]
2022-11-23T02:17:07.8056421Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_ykdllzxz)
2022-11-23T02:17:07.8056816Z 
2022-11-23T02:17:07.8057124Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-11-23 02:17:07.799647]
2022-11-23T02:17:07.8057853Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:07.800007]
2022-11-23T02:17:23.9214880Z 
2022-11-23T02:17:23.9215479Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-11-23T02:17:23.9216510Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_12s72d_u)
2022-11-23T02:17:23.9217597Z 
2022-11-23T02:17:23.9217734Z Running tests...
2022-11-23T02:17:23.9218589Z ----------------------------------------------------------------------
2022-11-23T02:17:23.9219253Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops
2022-11-23T02:17:23.9220161Z   test_clone (__main__.TestTensorOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:17:23.9221080Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45707
2022-11-23T02:17:23.9221954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45708
2022-11-23T02:17:23.9222816Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45709
2022-11-23T02:17:23.9223820Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45710
2022-11-23T02:17:23.9224616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9225054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9225632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9226104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9226661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9227106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9227679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9228141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9228699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9229149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9229880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9230344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9230924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9231366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9231935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9232379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9232813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:17:23.9233290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:17:23.9233757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9234200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9234588Z skip: Need at least 4 CUDA devices (4.080s)
2022-11-23T02:17:23.9235037Z   test_deep_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45843
2022-11-23T02:17:23.9235522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45844
2022-11-23T02:17:23.9235965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45845
2022-11-23T02:17:23.9236402Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45846
2022-11-23T02:17:23.9237077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9237630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9238175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9238627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9239201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9239649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9240233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9240691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9241264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9241693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9242250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9242689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9243256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9243699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9244273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9244729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9245148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:17:23.9245616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9246084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9246545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:17:23.9246975Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:17:23.9247430Z   test_detach (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45979
2022-11-23T02:17:23.9247928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45980
2022-11-23T02:17:23.9248353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45981
2022-11-23T02:17:23.9248788Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45982
2022-11-23T02:17:23.9249398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9249849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9250404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9250865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9251447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9251870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9252435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9252898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9253469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9253891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9254526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9254985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9255561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9255979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9256546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9257000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9257413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:17:23.9258124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9258735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9259210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:17:23.9259583Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:17:23.9260041Z   test_inplace_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46115
2022-11-23T02:17:23.9260541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46116
2022-11-23T02:17:23.9260964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46117
2022-11-23T02:17:23.9261403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46118
2022-11-23T02:17:23.9262018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9262472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9263031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9263498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9264175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9264610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9265187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9265651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9266225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9266641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9267207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9267671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9268247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9268666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9269234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9269691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9270105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:17:23.9270573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:17:23.9271028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9271578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9271948Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:17:23.9272407Z   test_set_requires_grad (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46251
2022-11-23T02:17:23.9272918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46252
2022-11-23T02:17:23.9273346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46253
2022-11-23T02:17:23.9273786Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46254
2022-11-23T02:17:23.9274387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9274842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9275392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9275863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9276434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9276878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9277427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9277886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9278457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9278876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9279440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9279903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9280477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9280952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9281527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9281983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9282399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:17:23.9282873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9283333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9283792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:17:23.9284165Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:17:23.9284356Z 
2022-11-23T02:17:23.9284630Z ----------------------------------------------------------------------
2022-11-23T02:17:23.9284962Z Ran 5 tests in 13.820s
2022-11-23T02:17:23.9285131Z 
2022-11-23T02:17:23.9285224Z OK (skipped=5)
2022-11-23T02:17:23.9285381Z 
2022-11-23T02:17:23.9285506Z Generating XML reports...
2022-11-23T02:17:23.9286128Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221123021709.xml
2022-11-23T02:17:23.9286492Z 
2022-11-23T02:17:23.9286821Z ##[endgroup]
2022-11-23T02:17:23.9287498Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_12s72d_u)
2022-11-23T02:17:23.9287891Z 
2022-11-23T02:17:23.9288211Z Running distributed/checkpoint/test_file_system_checkpoint_cpu ... [2022-11-23 02:17:23.921581]
2022-11-23T02:17:23.9289037Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:23.921914]
2022-11-23T02:17:40.0854431Z 
2022-11-23T02:17:40.0854996Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint_cpu
2022-11-23T02:17:40.0856051Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_4ck8t4tv)
2022-11-23T02:17:40.0857654Z 
2022-11-23T02:17:40.0858276Z Running tests...
2022-11-23T02:17:40.0859236Z ----------------------------------------------------------------------
2022-11-23T02:17:40.0860359Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu
2022-11-23T02:17:40.0861395Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:17:40.0862096Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46422
2022-11-23T02:17:40.0862537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46423
2022-11-23T02:17:40.0863162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0863615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0864233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0864685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0865264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0865713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0866285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0866727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0867341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:40.0867832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:40.0868297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:40.0868791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:40.0869450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0870141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0870521Z ok (3.967s)
2022-11-23T02:17:40.0870993Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46496
2022-11-23T02:17:40.0871561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46497
2022-11-23T02:17:40.0872170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0872595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0873158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0873624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0874178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0874732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0875298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0875762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0876177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:40.0876641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:40.0877120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:40.0877609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:40.0878238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0878929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0879323Z ok (2.509s)
2022-11-23T02:17:40.0879757Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46570
2022-11-23T02:17:40.0880296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46571
2022-11-23T02:17:40.0880896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0881336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0881890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0882349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0882923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0883366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0883916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0884428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0884870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:40.0885316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:40.0885793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:40.0886280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:40.0886930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0887598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0887986Z ok (2.307s)
2022-11-23T02:17:40.0888466Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46644
2022-11-23T02:17:40.0889016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46645
2022-11-23T02:17:40.0889621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0890064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0890633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0891084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0891770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0892215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0892784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0893228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0893657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:40.0894122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:40.0894584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:40.0895065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:40.0895723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0896400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0896784Z ok (2.609s)
2022-11-23T02:17:40.0897126Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.019s)
2022-11-23T02:17:40.0897761Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46718
2022-11-23T02:17:40.0898804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46719
2022-11-23T02:17:40.0899404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0899849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0900482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0900945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0901600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:40.0902057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:40.0902620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:40.0903061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:40.0903491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:40.0903955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:40.0904437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:40.0904908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:40.0905558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0906242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:40.0906631Z ok (2.408s)
2022-11-23T02:17:40.0906764Z 
2022-11-23T02:17:40.0907033Z ----------------------------------------------------------------------
2022-11-23T02:17:40.0907363Z Ran 6 tests in 13.821s
2022-11-23T02:17:40.0907524Z 
2022-11-23T02:17:40.0907617Z OK
2022-11-23T02:17:40.0907753Z 
2022-11-23T02:17:40.0907858Z Generating XML reports...
2022-11-23T02:17:40.0908546Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221123021725.xml
2022-11-23T02:17:40.0909576Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221123021725.xml
2022-11-23T02:17:40.0910598Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123021725.xml
2022-11-23T02:17:40.0911091Z 
2022-11-23T02:17:40.0911436Z ##[endgroup]
2022-11-23T02:17:40.0912127Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_4ck8t4tv)
2022-11-23T02:17:40.0912535Z 
2022-11-23T02:17:40.0912802Z Running distributed/fsdp/test_fsdp_apply ... [2022-11-23 02:17:40.085560]
2022-11-23T02:17:40.0913474Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:40.085898]
2022-11-23T02:17:56.5205702Z 
2022-11-23T02:17:56.5206251Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply
2022-11-23T02:17:56.5207780Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_fjf66o4q)
2022-11-23T02:17:56.5208151Z 
2022-11-23T02:17:56.5208249Z Running tests...
2022-11-23T02:17:56.5208759Z ----------------------------------------------------------------------
2022-11-23T02:17:56.5209318Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2022-11-23T02:17:56.5210015Z   test_apply_in_summon_raises_error (__main__.TestApply)
2022-11-23T02:17:56.5210707Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:17:56.5211696Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46827
2022-11-23T02:17:56.5212521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46828
2022-11-23T02:17:56.5213829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:56.5214758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:56.5215919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:56.5216409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:56.5216992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:56.5217418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:56.5218289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:56.5218764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:56.5219202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:56.5219696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:56.5220359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:56.5221044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:56.5221545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:56.5222010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:56.5223270Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:17:56.5224184Z   warnings.warn(
2022-11-23T02:17:56.5225334Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:17:56.5226106Z   warnings.warn(
2022-11-23T02:17:56.5226359Z   File "<string>", line 1, in <module>
2022-11-23T02:17:56.5226728Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:17:56.5227108Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:17:56.5227460Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:17:56.5227832Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:17:56.5228222Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:17:56.5228543Z     self.run()
2022-11-23T02:17:56.5228875Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:17:56.5229236Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:17:56.5229751Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:17:56.5230119Z     self.run_test(test_name, pipe)
2022-11-23T02:17:56.5230641Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:17:56.5231029Z     getattr(self, test_name)()
2022-11-23T02:17:56.5231530Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:17:56.5231896Z     fn()
2022-11-23T02:17:56.5232469Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:17:56.5232869Z     return func(*args, **kwargs)
2022-11-23T02:17:56.5233263Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_apply.py", line 98, in test_apply_in_summon_raises_error
2022-11-23T02:17:56.5233698Z     transformer.apply(self._init_linear_weights)
2022-11-23T02:17:56.5234266Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 492, in apply
2022-11-23T02:17:56.5234676Z     self._assert_state(TrainingState.IDLE)
2022-11-23T02:17:56.5235249Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1084, in _assert_state
2022-11-23T02:17:56.5235663Z     traceback.print_stack()
2022-11-23T02:17:56.5235910Z dist init r=0, world=2
2022-11-23T02:17:56.5236238Z Asserting FSDP instance is: FullyShardedDataParallel(
2022-11-23T02:17:56.5236632Z   (_fsdp_wrapped_module): TransformerWithSharedParams(
2022-11-23T02:17:56.5236975Z     (embed_tokens): Embedding(23, 16)
2022-11-23T02:17:56.5237245Z     (transformer): Transformer(
2022-11-23T02:17:56.5237536Z       (encoder): TransformerEncoder(
2022-11-23T02:17:56.5237816Z         (layers): ModuleList(
2022-11-23T02:17:56.5238091Z           (0): FullyShardedDataParallel(
2022-11-23T02:17:56.5238442Z             (_fsdp_wrapped_module): TransformerEncoderLayer(
2022-11-23T02:17:56.5238788Z               (self_attn): MultiheadAttention(
2022-11-23T02:17:56.5239181Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:17:56.5239538Z               )
2022-11-23T02:17:56.5239838Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-11-23T02:17:56.5240231Z               (dropout): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5240588Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-11-23T02:17:56.5241048Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5241506Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5241834Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5242162Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5242432Z             )
2022-11-23T02:17:56.5242636Z           )
2022-11-23T02:17:56.5242907Z           (1): FullyShardedDataParallel(
2022-11-23T02:17:56.5243265Z             (_fsdp_wrapped_module): TransformerEncoderLayer(
2022-11-23T02:17:56.5243593Z               (self_attn): MultiheadAttention(
2022-11-23T02:17:56.5244009Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:17:56.5244370Z               )
2022-11-23T02:17:56.5244660Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-11-23T02:17:56.5245007Z               (dropout): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5245362Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-11-23T02:17:56.5245822Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5246259Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5246607Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5246937Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5247190Z             )
2022-11-23T02:17:56.5247412Z           )
2022-11-23T02:17:56.5247625Z         )
2022-11-23T02:17:56.5247979Z         (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5248270Z       )
2022-11-23T02:17:56.5248525Z       (decoder): TransformerDecoder(
2022-11-23T02:17:56.5248799Z         (layers): ModuleList(
2022-11-23T02:17:56.5249094Z           (0): FullyShardedDataParallel(
2022-11-23T02:17:56.5249444Z             (_fsdp_wrapped_module): TransformerDecoderLayer(
2022-11-23T02:17:56.5249769Z               (self_attn): MultiheadAttention(
2022-11-23T02:17:56.5250239Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:17:56.5250601Z               )
2022-11-23T02:17:56.5250878Z               (multihead_attn): MultiheadAttention(
2022-11-23T02:17:56.5251276Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:17:56.5251623Z               )
2022-11-23T02:17:56.5251921Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-11-23T02:17:56.5252253Z               (dropout): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5252599Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-11-23T02:17:56.5253054Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5253487Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5253938Z               (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5254290Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5254617Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5254924Z               (dropout3): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5255192Z             )
2022-11-23T02:17:56.5255413Z           )
2022-11-23T02:17:56.5255663Z           (1): FullyShardedDataParallel(
2022-11-23T02:17:56.5256016Z             (_fsdp_wrapped_module): TransformerDecoderLayer(
2022-11-23T02:17:56.5256365Z               (self_attn): MultiheadAttention(
2022-11-23T02:17:56.5256761Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:17:56.5257180Z               )
2022-11-23T02:17:56.5257467Z               (multihead_attn): MultiheadAttention(
2022-11-23T02:17:56.5257866Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:17:56.5258424Z               )
2022-11-23T02:17:56.5258726Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-11-23T02:17:56.5259075Z               (dropout): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5259408Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-11-23T02:17:56.5259866Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5260313Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5260735Z               (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5261085Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5261418Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5261744Z               (dropout3): Dropout(p=0.1, inplace=False)
2022-11-23T02:17:56.5261999Z             )
2022-11-23T02:17:56.5262262Z           )
2022-11-23T02:17:56.5262482Z         )
2022-11-23T02:17:56.5262839Z         (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:17:56.5263132Z       )
2022-11-23T02:17:56.5263344Z     )
2022-11-23T02:17:56.5263638Z     (output_proj): Linear(in_features=16, out_features=23, bias=True)
2022-11-23T02:17:56.5264115Z     (bn): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
2022-11-23T02:17:56.5264436Z   )
2022-11-23T02:17:56.5264646Z )
2022-11-23T02:17:56.5265002Z ERROR: expected to be in states [<TrainingState.IDLE: 1>] but current state is TrainingState.SUMMON_FULL_PARAMS
2022-11-23T02:17:56.5265383Z dist init r=1, world=2
2022-11-23T02:17:56.5265626Z ok (5.627s)
2022-11-23T02:17:56.5265889Z   test_nested_module_apply (__main__.TestApply)
2022-11-23T02:17:56.5266508Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46906
2022-11-23T02:17:56.5267044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46907
2022-11-23T02:17:56.5267735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:56.5268169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:56.5268743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:56.5269211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:56.5269770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:56.5270215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:56.5270786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:56.5271248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:56.5271684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:56.5272178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:56.5272825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:56.5273495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:56.5274013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:56.5274476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:56.5275818Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:17:56.5276593Z   warnings.warn(
2022-11-23T02:17:56.5277713Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:17:56.5278484Z   warnings.warn(
2022-11-23T02:17:56.5278735Z dist init r=1, world=2
2022-11-23T02:17:56.5278986Z dist init r=0, world=2
2022-11-23T02:17:56.5279207Z ok (4.111s)
2022-11-23T02:17:56.5279499Z   test_transformer_module_apply (__main__.TestApply)
2022-11-23T02:17:56.5280129Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46985
2022-11-23T02:17:56.5280647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46986
2022-11-23T02:17:56.5281249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:56.5281693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:56.5282269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:56.5282722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:56.5283294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:56.5283736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:56.5284362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:56.5284811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:56.5285260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:56.5285747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:56.5286386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:56.5287069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:56.5287594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:56.5288067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:56.5289304Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:17:56.5290081Z   warnings.warn(
2022-11-23T02:17:56.5291229Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:17:56.5292096Z   warnings.warn(
2022-11-23T02:17:56.5292346Z dist init r=1, world=2
2022-11-23T02:17:56.5292577Z dist init r=0, world=2
2022-11-23T02:17:56.5292816Z ok (4.412s)
2022-11-23T02:17:56.5292965Z 
2022-11-23T02:17:56.5293237Z ----------------------------------------------------------------------
2022-11-23T02:17:56.5293548Z Ran 3 tests in 14.150s
2022-11-23T02:17:56.5293704Z 
2022-11-23T02:17:56.5293798Z OK
2022-11-23T02:17:56.5293933Z 
2022-11-23T02:17:56.5294065Z Generating XML reports...
2022-11-23T02:17:56.5294625Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123021741.xml
2022-11-23T02:17:56.5294940Z 
2022-11-23T02:17:56.5295284Z ##[endgroup]
2022-11-23T02:17:56.5295869Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_fjf66o4q)
2022-11-23T02:17:56.5296220Z 
2022-11-23T02:17:56.5296503Z Running distributed/_tensor/test_tp_sharding_ops ... [2022-11-23 02:17:56.520867]
2022-11-23T02:17:56.5297173Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tp_sharding_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:56.521177]
2022-11-23T02:18:15.2849442Z 
2022-11-23T02:18:15.2850466Z Expand the folded group to see the log file of distributed/_tensor/test_tp_sharding_ops
2022-11-23T02:18:15.2852095Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_zjhe64kx)
2022-11-23T02:18:15.2852759Z 
2022-11-23T02:18:15.2852960Z Running tests...
2022-11-23T02:18:15.2853710Z ----------------------------------------------------------------------
2022-11-23T02:18:15.2854289Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops
2022-11-23T02:18:15.2855495Z   test_replicated_permute (__main__.TPShardingOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:15.2856347Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47099
2022-11-23T02:18:15.2857185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47100
2022-11-23T02:18:15.2858311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47101
2022-11-23T02:18:15.2858995Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47102
2022-11-23T02:18:15.2859643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2860094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2860665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2861132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2861714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2862136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2862706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2863169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2863745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2864165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2864730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2865348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2865929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2866347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2866913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2867369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2867784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:15.2868251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:15.2868712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:15.2869177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:15.2869544Z skip: Need at least 4 CUDA devices (4.128s)
2022-11-23T02:18:15.2870009Z   test_sharded_cat (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47235
2022-11-23T02:18:15.2870520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47236
2022-11-23T02:18:15.2870944Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47237
2022-11-23T02:18:15.2871384Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47238
2022-11-23T02:18:15.2871991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2872439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2872996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2873466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2874045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2874561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2875149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2875612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2876183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2876605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2877173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2877638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2878211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2878635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2879202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2879666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2880080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:15.2880547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:15.2880999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:15.2881461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:15.2881897Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:18:15.2882364Z   test_sharded_permute (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47371
2022-11-23T02:18:15.2882884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47372
2022-11-23T02:18:15.2883311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47373
2022-11-23T02:18:15.2883747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47374
2022-11-23T02:18:15.2884409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2884858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2885426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2885877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2886451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2886894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2887462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2887904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2888474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2888910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2889458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2889918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2890492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2890925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2891581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2892052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2892489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:15.2892938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:15.2893405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:15.2893857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:15.2894257Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:18:15.2894701Z   test_sharded_split (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47507
2022-11-23T02:18:15.2895213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47508
2022-11-23T02:18:15.2895656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47509
2022-11-23T02:18:15.2896074Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47510
2022-11-23T02:18:15.2896685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2897133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2897706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2898362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2899044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2899485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2900049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2900492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2901063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2901501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2902049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2902508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2903083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2903521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2904068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2904528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2904960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:15.2905404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:15.2905865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:15.2906319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:15.2906710Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:18:15.2907165Z   test_sharded_transpose (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47643
2022-11-23T02:18:15.2907682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47644
2022-11-23T02:18:15.2908199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47645
2022-11-23T02:18:15.2908652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47646
2022-11-23T02:18:15.2909244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2909682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2910252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2910701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2911277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2911723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2912294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2912739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2913312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2913749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2914300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2914761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2915334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2915837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2916390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2916852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2917281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:15.2917743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:15.2918181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:15.2918633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:15.2919020Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:18:15.2919460Z   test_sharded_view (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47779
2022-11-23T02:18:15.2919975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47780
2022-11-23T02:18:15.2920414Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47781
2022-11-23T02:18:15.2920855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47782
2022-11-23T02:18:15.2921446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2921891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2922459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2922906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2923486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2923930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2924501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2925003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2925583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2926017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2926564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2927025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2927597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:15.2928035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:15.2928581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:15.2929043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:15.2929475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:15.2929939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:15.2930382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:15.2930833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:15.2931220Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:18:15.2931416Z 
2022-11-23T02:18:15.2931670Z ----------------------------------------------------------------------
2022-11-23T02:18:15.2932061Z Ran 6 tests in 16.380s
2022-11-23T02:18:15.2932224Z 
2022-11-23T02:18:15.2932335Z OK (skipped=6)
2022-11-23T02:18:15.2932494Z 
2022-11-23T02:18:15.2932621Z Generating XML reports...
2022-11-23T02:18:15.2933210Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20221123021758.xml
2022-11-23T02:18:15.2933570Z 
2022-11-23T02:18:15.2933906Z ##[endgroup]
2022-11-23T02:18:15.2934521Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_zjhe64kx)
2022-11-23T02:18:15.2934865Z 
2022-11-23T02:18:15.2935152Z Running distributed/_tensor/parallel/test_tp_style ... [2022-11-23 02:18:15.285080]
2022-11-23T02:18:15.2935854Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_tp_style.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:15.285451]
2022-11-23T02:18:38.8746376Z 
2022-11-23T02:18:38.8747272Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_tp_style
2022-11-23T02:18:38.8748634Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_style (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_tp_style_uio9avax)
2022-11-23T02:18:38.8750888Z 
2022-11-23T02:18:38.8751569Z Running tests...
2022-11-23T02:18:38.8752123Z ----------------------------------------------------------------------
2022-11-23T02:18:38.8752718Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_tp_style
2022-11-23T02:18:38.8753280Z   test_colwise_parallel_style (__main__.TensorParallelStyleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:38.8753755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47950
2022-11-23T02:18:38.8754206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47951
2022-11-23T02:18:38.8754641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47952
2022-11-23T02:18:38.8755099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47953
2022-11-23T02:18:38.8755702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8756420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8759793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8760292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8760912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8761367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8761945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8762414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8762994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8763437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8763992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8764463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8765044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8765490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8766049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8766763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8767294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:38.8767770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:38.8768237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:38.8768705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:38.8769082Z skip: Need at least 4 CUDA devices (4.046s)
2022-11-23T02:18:38.8769574Z   test_make_input_replicate_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48086
2022-11-23T02:18:38.8770121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48087
2022-11-23T02:18:38.8770551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48088
2022-11-23T02:18:38.8770991Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48089
2022-11-23T02:18:38.8771609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8772064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8772619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8773085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8773656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8774077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8774645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8775106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8775694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8776116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8776764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8777237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8777789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8778614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8779190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8779646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8780066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:38.8780535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:38.8781008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:38.8781467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:38.8781839Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:18:38.8782322Z   test_make_input_shard_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48222
2022-11-23T02:18:38.8782857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48223
2022-11-23T02:18:38.8783281Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48224
2022-11-23T02:18:38.8783715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48225
2022-11-23T02:18:38.8784436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8784884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8785446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8785916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8786496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8786917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8787487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8787951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8788529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8788950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8789524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8789987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8790560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8790983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8791600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8792058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8792472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:38.8792943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:38.8793404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:38.8793947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:38.8794335Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:18:38.8794829Z   test_make_output_replicate_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48358
2022-11-23T02:18:38.8795367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48359
2022-11-23T02:18:38.8795789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48360
2022-11-23T02:18:38.8796227Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48361
2022-11-23T02:18:38.8796840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8797288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8797844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8798308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8798886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8799333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8799887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8800349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8800923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8801404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8801982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8802440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8803015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8803436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8804001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8804457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8804867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:38.8805342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:38.8805798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:38.8806263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:38.8806631Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:18:38.8807115Z   test_make_output_shard_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48494
2022-11-23T02:18:38.8807649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48495
2022-11-23T02:18:38.8808073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48496
2022-11-23T02:18:38.8808513Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48497
2022-11-23T02:18:38.8809117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8809569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8810120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8810640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8811224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8811664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8812211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8812668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8813236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8813655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8814223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8814681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8815250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8815665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8816232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8816687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8817095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:38.8817562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:38.8818348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:38.8818816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:38.8819188Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:18:38.8819671Z   test_make_output_tensor (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48630
2022-11-23T02:18:38.8820201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48631
2022-11-23T02:18:38.8820646Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48632
2022-11-23T02:18:38.8821069Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48633
2022-11-23T02:18:38.8821678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8822133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8822685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8823157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8823737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8824180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8824728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8825181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8825754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8826173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8826748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8827207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8827861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8828296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8828868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8829326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8829758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:38.8830206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:38.8830666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:38.8831127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:38.8831496Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:18:38.8831988Z   test_prepare_output_error (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48766
2022-11-23T02:18:38.8832523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48767
2022-11-23T02:18:38.8832965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48768
2022-11-23T02:18:38.8833386Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48769
2022-11-23T02:18:38.8833986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8834433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8835072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8835542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8836120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8836563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8837102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8837542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8838107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8838554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8839138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8839596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8840171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8840591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8841159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8841620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8842052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:38.8842500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:38.8842958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:38.8843423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:38.8843791Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:18:38.8844335Z   test_rowwise_parallel_style (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48902
2022-11-23T02:18:38.8844889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48903
2022-11-23T02:18:38.8845332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48904
2022-11-23T02:18:38.8845749Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48905
2022-11-23T02:18:38.8846351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8846795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8847349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8847818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8848395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8848836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8849382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8849846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8850419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8850855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8851404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8851924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8852502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:38.8852919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:38.8853488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:38.8853952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:38.8854384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:38.8854832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:38.8855285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:38.8855751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:38.8856122Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:18:38.8856317Z 
2022-11-23T02:18:38.8856597Z ----------------------------------------------------------------------
2022-11-23T02:18:38.8856929Z Ran 8 tests in 21.215s
2022-11-23T02:18:38.8857096Z 
2022-11-23T02:18:38.8857206Z OK (skipped=8)
2022-11-23T02:18:38.8857342Z 
2022-11-23T02:18:38.8857468Z Generating XML reports...
2022-11-23T02:18:38.8858438Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_tp_style/TEST-TensorParallelStyleTest-20221123021817.xml
2022-11-23T02:18:38.8858825Z 
2022-11-23T02:18:38.8859284Z ##[endgroup]
2022-11-23T02:18:38.8859912Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_style (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_tp_style_uio9avax)
2022-11-23T02:18:38.8860290Z 
2022-11-23T02:18:38.8860606Z Running distributed/checkpoint/test_file_system_checkpoint ... [2022-11-23 02:18:38.874914]
2022-11-23T02:18:38.8861436Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:38.875193]
2022-11-23T02:19:04.1159973Z 
2022-11-23T02:19:04.1160700Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint
2022-11-23T02:19:04.1161690Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_0j_rc7ij)
2022-11-23T02:19:04.1162105Z 
2022-11-23T02:19:04.1162216Z Running tests...
2022-11-23T02:19:04.1168318Z ----------------------------------------------------------------------
2022-11-23T02:19:04.1168991Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint
2022-11-23T02:19:04.1169568Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:19:04.1170069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49073
2022-11-23T02:19:04.1170521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49074
2022-11-23T02:19:04.1171156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1171590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1172228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1172700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1173301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1174007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1186844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1187415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1187871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:04.1188344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:04.1188804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:04.1189305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:04.1189996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1191333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1192135Z ok (5.663s)
2022-11-23T02:19:04.1193068Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49156
2022-11-23T02:19:04.1194098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49157
2022-11-23T02:19:04.1195302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1196287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1197320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1197796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1198354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1198805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1199375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1200005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1200436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:04.1200919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:04.1201394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:04.1201907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:04.1202566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1203235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1203625Z ok (4.514s)
2022-11-23T02:19:04.1204076Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49239
2022-11-23T02:19:04.1204599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49240
2022-11-23T02:19:04.1205200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1205643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1206214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1206659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1207225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1207770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1208340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1208784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1209214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:04.1209695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:04.1210156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:04.1210636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:04.1211284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1211969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1212340Z ok (3.911s)
2022-11-23T02:19:04.1212820Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49322
2022-11-23T02:19:04.1213385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49323
2022-11-23T02:19:04.1213984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1214411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1214976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1215439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1215997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1216437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1217057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1217525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1218455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:04.1218975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:04.1219453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:04.1219909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:04.1220578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1221253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1221647Z ok (4.713s)
2022-11-23T02:19:04.1221988Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.056s)
2022-11-23T02:19:04.1222629Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49405
2022-11-23T02:19:04.1223236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49406
2022-11-23T02:19:04.1223838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1224266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1224942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1225404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1225959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:04.1226397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:04.1226959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:04.1227421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:04.1227835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:04.1228296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:04.1228777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:04.1229250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:04.1229901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1230582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:04.1230973Z ok (4.011s)
2022-11-23T02:19:04.1231123Z 
2022-11-23T02:19:04.1231376Z ----------------------------------------------------------------------
2022-11-23T02:19:04.1231705Z Ran 6 tests in 22.868s
2022-11-23T02:19:04.1231860Z 
2022-11-23T02:19:04.1231955Z OK
2022-11-23T02:19:04.1232085Z 
2022-11-23T02:19:04.1232210Z Generating XML reports...
2022-11-23T02:19:04.1232873Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123021840.xml
2022-11-23T02:19:04.1233802Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123021840.xml
2022-11-23T02:19:04.1234879Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123021840.xml
2022-11-23T02:19:04.1235370Z 
2022-11-23T02:19:04.1235806Z ##[endgroup]
2022-11-23T02:19:04.1236465Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_0j_rc7ij)
2022-11-23T02:19:04.1236866Z 
2022-11-23T02:19:04.1237145Z Running distributed/checkpoint/test_checkpoint ... [2022-11-23 02:19:04.116241]
2022-11-23T02:19:04.1237838Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:19:04.116566]
2022-11-23T02:19:30.2969148Z 
2022-11-23T02:19:30.2969673Z Expand the folded group to see the log file of distributed/checkpoint/test_checkpoint
2022-11-23T02:19:30.2975064Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_mf7603d4)
2022-11-23T02:19:30.2975774Z 
2022-11-23T02:19:30.2975923Z Running tests...
2022-11-23T02:19:30.2977753Z ----------------------------------------------------------------------
2022-11-23T02:19:30.2979189Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_checkpoint
2022-11-23T02:19:30.2980008Z   test_default_metadata (__main__.TestDistributedCheckpointing) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:19:30.2990357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49523
2022-11-23T02:19:30.2990977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49524
2022-11-23T02:19:30.2991932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.2992395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.2992962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.2993465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.2994050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.2994507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.2995060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.2995530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.2995978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:30.2996464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:30.2996931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:30.2997420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:30.2998076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:30.2998760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:30.2999128Z ok (5.524s)
2022-11-23T02:19:30.2999614Z   test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49602
2022-11-23T02:19:30.3000188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49603
2022-11-23T02:19:30.3000776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3001342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3001956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3002428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3002985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3003433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3004004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3004475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3004898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:30.3005374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:30.3005854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:30.3006344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:30.3006981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:30.3007700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:30.3008097Z ok (3.910s)
2022-11-23T02:19:30.3008541Z   test_dummy_reader_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49681
2022-11-23T02:19:30.3009126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49682
2022-11-23T02:19:30.3009569Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49683
2022-11-23T02:19:30.3010020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49684
2022-11-23T02:19:30.3010609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3011061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3011633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3012109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3012666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3013119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3013694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3014159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3014716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3015153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3015721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3016160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3016732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3017167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3017738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3018637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3019171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:30.3019653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:30.3020095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:30.3020556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:30.3020944Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:19:30.3021426Z   test_dummy_writer_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49817
2022-11-23T02:19:30.3021938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49818
2022-11-23T02:19:30.3022377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49819
2022-11-23T02:19:30.3022810Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49820
2022-11-23T02:19:30.3023409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3023859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3024432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3024895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3025449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3025889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3026555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3027019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3027573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3028013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3028578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3029014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3029586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3030028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3030597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3031035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3031528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:30.3031998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:30.3032466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:30.3032925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:30.3033297Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:19:30.3033777Z   test_load_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49953
2022-11-23T02:19:30.3034301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49954
2022-11-23T02:19:30.3034725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49955
2022-11-23T02:19:30.3035152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49956
2022-11-23T02:19:30.3035816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3036275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3036827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3037295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3037870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3038291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3038860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3039326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3039902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3040320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3040887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3041343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3041895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3042332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3042900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3043423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3043835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:30.3044305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:30.3044772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:30.3045232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:30.3045602Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:19:30.3046091Z   test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50089
2022-11-23T02:19:30.3046626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50090
2022-11-23T02:19:30.3047056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50091
2022-11-23T02:19:30.3047487Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50092
2022-11-23T02:19:30.3048098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3048547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3049085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3049522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3050090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3050533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3051109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3051575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3052142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3052624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3053206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3053665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3054238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3054659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3055220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3055683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3056096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:30.3056561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:30.3057022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:30.3057478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:30.3057802Z ok (2.410s)
2022-11-23T02:19:30.3058678Z   test_save_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50225
2022-11-23T02:19:30.3059211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50226
2022-11-23T02:19:30.3059636Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50227
2022-11-23T02:19:30.3060190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50228
2022-11-23T02:19:30.3060797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3061246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3061800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3062268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3062848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3063268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3063836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3064295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3064873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3065291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3065859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3066318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3066888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3067305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3067872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3068374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3068792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:30.3069261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:30.3069800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:30.3070279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:30.3070649Z skip: Need at least 4 CUDA devices (2.420s)
2022-11-23T02:19:30.3071138Z   test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50361
2022-11-23T02:19:30.3071675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50362
2022-11-23T02:19:30.3072100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50363
2022-11-23T02:19:30.3072535Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50364
2022-11-23T02:19:30.3073142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3073589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3074144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3074610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3075185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3075628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3076161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3076600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3077256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3077700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3078284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3078750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3079318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:30.3079741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:30.3080307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:30.3080764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:30.3081179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:30.3081649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:30.3082105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:30.3082566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:30.3082889Z ok (2.410s)
2022-11-23T02:19:30.3083039Z 
2022-11-23T02:19:30.3083315Z ----------------------------------------------------------------------
2022-11-23T02:19:30.3083646Z Ran 8 tests in 23.904s
2022-11-23T02:19:30.3083810Z 
2022-11-23T02:19:30.3083902Z OK (skipped=4)
2022-11-23T02:19:30.3084056Z 
2022-11-23T02:19:30.3084179Z Generating XML reports...
2022-11-23T02:19:30.3084837Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123021905.xml
2022-11-23T02:19:30.3085679Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123021905.xml
2022-11-23T02:19:30.3086059Z 
2022-11-23T02:19:30.3086505Z ##[endgroup]
2022-11-23T02:19:30.3087214Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_mf7603d4)
2022-11-23T02:19:30.3087599Z 
2022-11-23T02:19:30.3087878Z Running distributed/_tensor/test_redistribute ... [2022-11-23 02:19:30.297172]
2022-11-23T02:19:30.3088563Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:19:30.297454]
2022-11-23T02:20:01.0851797Z 
2022-11-23T02:20:01.0852698Z Expand the folded group to see the log file of distributed/_tensor/test_redistribute
2022-11-23T02:20:01.0853845Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_vbvnn504)
2022-11-23T02:20:01.0854293Z 
2022-11-23T02:20:01.0854412Z Running tests...
2022-11-23T02:20:01.0857067Z ----------------------------------------------------------------------
2022-11-23T02:20:01.0857680Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_redistribute
2022-11-23T02:20:01.0858475Z   test_multi_dim_mesh (__main__.MultiDimRedistributeTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:20:01.0858964Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50532
2022-11-23T02:20:01.0861839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50533
2022-11-23T02:20:01.0862599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50534
2022-11-23T02:20:01.0863062Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50535
2022-11-23T02:20:01.0863738Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 50536
2022-11-23T02:20:01.0864443Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 50537
2022-11-23T02:20:01.0865137Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 50538
2022-11-23T02:20:01.0865612Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 50539
2022-11-23T02:20:01.0866537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0867001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0867801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0868323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0869150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0869631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0870335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0870953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0871634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0872245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0872836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0873557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0874127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0874847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0875435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0876173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0876872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0877616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0878217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0878956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0879532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0880231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0880817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0881501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0882110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0882726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0883440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0884038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0884721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0885251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0885977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0886630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0887234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:20:01.0887718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:20:01.0888508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:01.0888974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:01.0889735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:20:01.0890208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:01.0890911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:01.0891410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:20:01.0891791Z skip: Need at least 8 CUDA devices (4.225s)
2022-11-23T02:20:01.0892543Z   test_partial_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50804
2022-11-23T02:20:01.0893093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50805
2022-11-23T02:20:01.0893963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0894415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0895247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0895722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0896528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0896988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0897830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0898943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0899496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:01.0900005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:01.0900725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:01.0901190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:01.0902132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0903012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0903493Z ok (3.911s)
2022-11-23T02:20:01.0903904Z   test_partial_to_shard (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50887
2022-11-23T02:20:01.0904671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50888
2022-11-23T02:20:01.0905292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0905976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0906561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0907289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0907880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0908684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0909276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0910001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0910427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:01.0911084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:01.0911651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:01.0912166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:01.0913029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0913974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0914381Z ok (4.011s)
2022-11-23T02:20:01.0914835Z   test_replicate_to_partial (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50966
2022-11-23T02:20:01.0915572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50967
2022-11-23T02:20:01.0916250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0916897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0917460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0918091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0918789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0919240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0919866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0920452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0921058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:01.0921547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:01.0922017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:01.0922501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:01.0923170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0923835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0924437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:20:01.0925131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:20:01.0925791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:20:01.0926454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:20:01.0926980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:20:01.0927469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:20:01.0928198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:20:01.0928859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:20:01.0929381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T02:20:01.0929860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T02:20:01.0930500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:20:01.0931156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:20:01.0931548Z ok (4.112s)
2022-11-23T02:20:01.0932286Z   test_replicate_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51064
2022-11-23T02:20:01.0932819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51065
2022-11-23T02:20:01.0933439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0933890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0934465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0934916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0935497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0935938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0936496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0936963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0937465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:01.0938363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:01.0938841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:01.0939318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:01.0939981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0940672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0941049Z ok (4.011s)
2022-11-23T02:20:01.0941493Z   test_replicate_to_shard_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51147
2022-11-23T02:20:01.0942040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51148
2022-11-23T02:20:01.0942630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0943076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0943648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0944107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0944657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0945104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0946150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0946624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0947046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:01.0947532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:01.0948011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:01.0948469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:01.0949122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0949808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0950212Z ok (4.111s)
2022-11-23T02:20:01.0950640Z   test_shard_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51230
2022-11-23T02:20:01.0951184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51231
2022-11-23T02:20:01.0951792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0952243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0952794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0953255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0953828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:01.0954254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:01.0954823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:01.0955376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:01.0955827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:01.0956290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:01.0956769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:01.0957249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:01.0957877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0958568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:01.0958963Z ok (4.011s)
2022-11-23T02:20:01.0959115Z 
2022-11-23T02:20:01.0959387Z ----------------------------------------------------------------------
2022-11-23T02:20:01.0959706Z Ran 7 tests in 28.394s
2022-11-23T02:20:01.0959871Z 
2022-11-23T02:20:01.0959983Z OK (skipped=1)
2022-11-23T02:20:01.0960136Z 
2022-11-23T02:20:01.0960259Z Generating XML reports...
2022-11-23T02:20:01.0960831Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20221123021932.xml
2022-11-23T02:20:01.0961620Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20221123021932.xml
2022-11-23T02:20:01.0961985Z 
2022-11-23T02:20:01.0962425Z ##[endgroup]
2022-11-23T02:20:01.0963042Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_vbvnn504)
2022-11-23T02:20:01.0963480Z 
2022-11-23T02:20:01.0963741Z Running distributed/_tensor/test_common_rules ... [2022-11-23 02:20:01.085304]
2022-11-23T02:20:01.0964436Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:20:01.085574]
2022-11-23T02:20:32.0858933Z 
2022-11-23T02:20:32.0859901Z Expand the folded group to see the log file of distributed/_tensor/test_common_rules
2022-11-23T02:20:32.0861215Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_k4t7vjfd)
2022-11-23T02:20:32.0861586Z 
2022-11-23T02:20:32.0861704Z Running tests...
2022-11-23T02:20:32.0862232Z ----------------------------------------------------------------------
2022-11-23T02:20:32.0862791Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_common_rules
2022-11-23T02:20:32.0866612Z   test_einop_basic_propagation (__main__.CommonRulesTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:20:32.0867707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51350
2022-11-23T02:20:32.0868323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51351
2022-11-23T02:20:32.0868769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51352
2022-11-23T02:20:32.0869196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51353
2022-11-23T02:20:32.0869883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0870337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0870921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0871380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0871965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0872690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0873762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0874663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0875357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0875806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0876621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0877390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0878550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0879321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0879892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0880360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0880797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0881345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0881819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0882274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0882785Z skip: Need at least 4 CUDA devices (4.100s)
2022-11-23T02:20:32.0883279Z   test_einop_errors (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51486
2022-11-23T02:20:32.0883779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51487
2022-11-23T02:20:32.0884210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51488
2022-11-23T02:20:32.0884662Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51489
2022-11-23T02:20:32.0885319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0886120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0887291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0887998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0888595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0889040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0889591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0890052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0890674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0891093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0891660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0892119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0892694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0893116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0893760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0894229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0894645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0895113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0895564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0896024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0896394Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:20:32.0896851Z   test_einop_linearity (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51622
2022-11-23T02:20:32.0897361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51623
2022-11-23T02:20:32.0897812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51624
2022-11-23T02:20:32.0898615Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51625
2022-11-23T02:20:32.0899226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0899674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0900226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0900689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0901263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0901813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0902362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0902827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0903398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0903818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0904383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0904843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0905411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0905837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0906405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0906866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0907279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0907749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0908204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0908662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0909041Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:20:32.0909505Z   test_einop_merge_sharding (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51758
2022-11-23T02:20:32.0910023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51759
2022-11-23T02:20:32.0910466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51760
2022-11-23T02:20:32.0910958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51761
2022-11-23T02:20:32.0911568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0912014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0912564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0913036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0913610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0914054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0914600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0915063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0915633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0916052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0916617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0917073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0917640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0918058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0918686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0919144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0919580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0920029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0920485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0920943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0921313Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:20:32.0921794Z   test_einop_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51894
2022-11-23T02:20:32.0922326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51895
2022-11-23T02:20:32.0922769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51896
2022-11-23T02:20:32.0923190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51897
2022-11-23T02:20:32.0923791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0924236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0924784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0925247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0925818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0926255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0926808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0927268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0927890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0928338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0928883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0929339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0929904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0930320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0930890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0931349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0931782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0932233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0932697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0933156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0933524Z skip: Need at least 4 CUDA devices (2.512s)
2022-11-23T02:20:32.0934005Z   test_einop_pointwise_propagation (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52030
2022-11-23T02:20:32.0934535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52031
2022-11-23T02:20:32.0935042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52032
2022-11-23T02:20:32.0935460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52033
2022-11-23T02:20:32.0936061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0936511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0937060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0937525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0938394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0938843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0939401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0939865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0940440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0940885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0941434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0941893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0942462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0942879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0943445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0943905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0944337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0944860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0945327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0945788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0946160Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:20:32.0946668Z   test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52166
2022-11-23T02:20:32.0947217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52167
2022-11-23T02:20:32.0947666Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52168
2022-11-23T02:20:32.0948090Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52169
2022-11-23T02:20:32.0948696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0949139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0949710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0950156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0950729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0951170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0951717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0952259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0952831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0953270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0953815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0954275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0954850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0955271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0955837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0956300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0956735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0957182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0957641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0958101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0958492Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:20:32.0958956Z   test_pointwise_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52302
2022-11-23T02:20:32.0959488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52303
2022-11-23T02:20:32.0959937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52304
2022-11-23T02:20:32.0960357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52305
2022-11-23T02:20:32.0961011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0961464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0962030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0962476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0963045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0963487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0964035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0964502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0965070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0965512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0966058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0966517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0967084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0967525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0968072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0968591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0969022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0969471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0969922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0970385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0970773Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:20:32.0971229Z   test_pointwise_rules_broadcasting (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52438
2022-11-23T02:20:32.0971759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52439
2022-11-23T02:20:32.0972203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52440
2022-11-23T02:20:32.0972631Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52441
2022-11-23T02:20:32.0973235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0973682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0974252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0974699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0975269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0975704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0976248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0976713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0977281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0977717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0978732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0979202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0979770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0980208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0980749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0981210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0981651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0982097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0982566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0983021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0983406Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:20:32.0983859Z   test_pointwise_rules_suggestion (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52574
2022-11-23T02:20:32.0984382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52575
2022-11-23T02:20:32.0984865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52576
2022-11-23T02:20:32.0985310Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52577
2022-11-23T02:20:32.0985980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0986428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0986988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0987435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0987980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0988443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0989021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0989462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0990043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0990528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0991092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0991530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0992097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0992538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0993088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0993543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.0993984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.0994450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.0994950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.0995410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.0995799Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:20:32.0996234Z   test_reduction_rule (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52710
2022-11-23T02:20:32.0996737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52711
2022-11-23T02:20:32.0997180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52712
2022-11-23T02:20:32.0997620Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52713
2022-11-23T02:20:32.0998211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.0998653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.0999224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.0999691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.1000248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.1000689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.1001249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.1001684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.1002248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.1002745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.1003316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.1003752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.1004320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:32.1004756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:32.1005301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:32.1005754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:32.1006185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:32.1006656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:32.1007100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:32.1007555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:32.1007946Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:20:32.1008139Z 
2022-11-23T02:20:32.1008412Z ----------------------------------------------------------------------
2022-11-23T02:20:32.1008724Z Ran 11 tests in 28.608s
2022-11-23T02:20:32.1008888Z 
2022-11-23T02:20:32.1009001Z OK (skipped=11)
2022-11-23T02:20:32.1009157Z 
2022-11-23T02:20:32.1009282Z Generating XML reports...
2022-11-23T02:20:32.1009849Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20221123022003.xml
2022-11-23T02:20:32.1010191Z 
2022-11-23T02:20:32.1010646Z ##[endgroup]
2022-11-23T02:20:32.1011260Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_k4t7vjfd)
2022-11-23T02:20:32.1011610Z 
2022-11-23T02:20:32.1011868Z Running distributed/test_c10d_common ... [2022-11-23 02:20:32.086223]
2022-11-23T02:20:32.1012598Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:20:32.086490]
2022-11-23T02:21:38.4361501Z 
2022-11-23T02:21:38.4362252Z Expand the folded group to see the log file of distributed/test_c10d_common
2022-11-23T02:21:38.4363183Z ##[group]PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_voatt834)
2022-11-23T02:21:38.4363721Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_debug_level>]>
2022-11-23T02:21:38.4366408Z test_debug_level (__main__.CommTest)
2022-11-23T02:21:38.4368063Z <unittest.suite.TestSuite tests=[<__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_single_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_single_dtype>]>
2022-11-23T02:21:38.4368998Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T02:21:38.4369415Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T02:21:38.4370122Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T02:21:38.4370715Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T02:21:38.4372358Z <unittest.suite.TestSuite tests=[<__main__.PythonProcessGroupExtensionTest testMethod=test_backend_class_attr>, <__main__.PythonProcessGroupExtensionTest testMethod=test_collectives>, <__main__.PythonProcessGroupExtensionTest testMethod=test_get_backend_name>, <__main__.PythonProcessGroupExtensionTest testMethod=test_send_recv>]>
2022-11-23T02:21:38.4373495Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-11-23T02:21:38.4373909Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-11-23T02:21:38.4374337Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-11-23T02:21:38.4375050Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-11-23T02:21:38.4375937Z <unittest.suite.TestSuite tests=[<__main__.ReduceOpTest testMethod=test_op_isinstance_of_reduceop>, <__main__.ReduceOpTest testMethod=test_reduceop_copyable>, <__main__.ReduceOpTest testMethod=test_reduceop_pickle>]>
2022-11-23T02:21:38.4377101Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest)
2022-11-23T02:21:38.4377730Z test_reduceop_copyable (__main__.ReduceOpTest)
2022-11-23T02:21:38.4378822Z test_reduceop_pickle (__main__.ReduceOpTest)
2022-11-23T02:21:38.4379990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4380754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4381763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4382731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4383108Z 
2022-11-23T02:21:38.4383285Z Running tests...
2022-11-23T02:21:38.4383938Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4384803Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4385267Z   test_debug_level (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4385701Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52914
2022-11-23T02:21:38.4386120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52915
2022-11-23T02:21:38.4386725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4387169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4387860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4388346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4388921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4389356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4389905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4390411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4390849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:38.4391297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:38.4391636Z ok (3.949s)
2022-11-23T02:21:38.4391784Z 
2022-11-23T02:21:38.4392061Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4392386Z Ran 1 test in 3.950s
2022-11-23T02:21:38.4392526Z 
2022-11-23T02:21:38.4392619Z OK
2022-11-23T02:21:38.4392751Z 
2022-11-23T02:21:38.4392875Z Generating XML reports...
2022-11-23T02:21:38.4393417Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221123022035.xml
2022-11-23T02:21:38.4394045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4394488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4395052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4395602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4395828Z 
2022-11-23T02:21:38.4395920Z Running tests...
2022-11-23T02:21:38.4396320Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4396840Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4397330Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4397708Z ok (1.658s)
2022-11-23T02:21:38.4397855Z 
2022-11-23T02:21:38.4398118Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4398444Z Ran 1 test in 1.658s
2022-11-23T02:21:38.4398583Z 
2022-11-23T02:21:38.4398674Z OK
2022-11-23T02:21:38.4398805Z 
2022-11-23T02:21:38.4398931Z Generating XML reports...
2022-11-23T02:21:38.4399541Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123022042.xml
2022-11-23T02:21:38.4400249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4400697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4401260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4401722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4401944Z 
2022-11-23T02:21:38.4402035Z Running tests...
2022-11-23T02:21:38.4402430Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4402951Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4403441Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4403822Z ok (1.652s)
2022-11-23T02:21:38.4403969Z 
2022-11-23T02:21:38.4404231Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4404549Z Ran 1 test in 1.652s
2022-11-23T02:21:38.4404689Z 
2022-11-23T02:21:38.4404780Z OK
2022-11-23T02:21:38.4404977Z 
2022-11-23T02:21:38.4405106Z Generating XML reports...
2022-11-23T02:21:38.4405719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123022046.xml
2022-11-23T02:21:38.4406420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4406856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4407421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4407881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4408110Z 
2022-11-23T02:21:38.4408201Z Running tests...
2022-11-23T02:21:38.4408596Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4409114Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4409611Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4409993Z ok (1.661s)
2022-11-23T02:21:38.4410138Z 
2022-11-23T02:21:38.4410397Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4410715Z Ran 1 test in 1.661s
2022-11-23T02:21:38.4410855Z 
2022-11-23T02:21:38.4410948Z OK
2022-11-23T02:21:38.4411080Z 
2022-11-23T02:21:38.4411203Z Generating XML reports...
2022-11-23T02:21:38.4411813Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123022049.xml
2022-11-23T02:21:38.4412510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4413024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4413596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4414056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4414283Z 
2022-11-23T02:21:38.4414374Z Running tests...
2022-11-23T02:21:38.4414768Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4415288Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4415785Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4416166Z ok (1.653s)
2022-11-23T02:21:38.4416313Z 
2022-11-23T02:21:38.4416573Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4416900Z Ran 1 test in 1.653s
2022-11-23T02:21:38.4417041Z 
2022-11-23T02:21:38.4417135Z OK
2022-11-23T02:21:38.4417266Z 
2022-11-23T02:21:38.4417387Z Generating XML reports...
2022-11-23T02:21:38.4418613Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123022053.xml
2022-11-23T02:21:38.4419347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4419789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4420352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4420812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4421038Z 
2022-11-23T02:21:38.4421129Z Running tests...
2022-11-23T02:21:38.4421525Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4422055Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4422576Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4423167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53153
2022-11-23T02:21:38.4423625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53154
2022-11-23T02:21:38.4424063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53155
2022-11-23T02:21:38.4424478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53156
2022-11-23T02:21:38.4425074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4425516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4426085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4426534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4427108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4427545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4428078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4428518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4429077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4429532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4430088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4430632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4431198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4431621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4432176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4432630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4433057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:21:38.4433508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:38.4433957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:38.4434413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:21:38.4434758Z ok (4.018s)
2022-11-23T02:21:38.4434888Z 
2022-11-23T02:21:38.4435154Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4435474Z Ran 1 test in 4.018s
2022-11-23T02:21:38.4435635Z 
2022-11-23T02:21:38.4435728Z OK
2022-11-23T02:21:38.4435841Z 
2022-11-23T02:21:38.4435965Z Generating XML reports...
2022-11-23T02:21:38.4436597Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123022057.xml
2022-11-23T02:21:38.4437331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4437770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4438318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4438782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4439010Z 
2022-11-23T02:21:38.4439118Z Running tests...
2022-11-23T02:21:38.4439496Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4440082Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4440603Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4441100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53324
2022-11-23T02:21:38.4441522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53325
2022-11-23T02:21:38.4441954Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53326
2022-11-23T02:21:38.4442387Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53327
2022-11-23T02:21:38.4442975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4443421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4443983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4444446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4444997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4445433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4445980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4446398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4446962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4447488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4448066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4448509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4449079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4449514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4450072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4450508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4450936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:38.4451401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:21:38.4452013Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T02:21:38.4452671Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T02:21:38.4453159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:21:38.4453769Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T02:21:38.4454240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:38.4454717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:38.4455204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:21:38.4455686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:21:38.4456150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:38.4456850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:21:38.4457541Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:21:38.4458853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:21:38.4459539Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:21:38.4459926Z ok (6.722s)
2022-11-23T02:21:38.4460075Z 
2022-11-23T02:21:38.4460338Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4460649Z Ran 1 test in 6.722s
2022-11-23T02:21:38.4460809Z 
2022-11-23T02:21:38.4460901Z OK
2022-11-23T02:21:38.4461031Z 
2022-11-23T02:21:38.4461154Z Generating XML reports...
2022-11-23T02:21:38.4461773Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123022104.xml
2022-11-23T02:21:38.4462504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4462949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4463511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4463960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4464186Z 
2022-11-23T02:21:38.4464294Z Running tests...
2022-11-23T02:21:38.4464691Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4465322Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4465825Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4466409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53504
2022-11-23T02:21:38.4466834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53505
2022-11-23T02:21:38.4467263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53506
2022-11-23T02:21:38.4467697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53507
2022-11-23T02:21:38.4468279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4468721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4469285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4469752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4470306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4470745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4471305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4471745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4472310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4472744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4473304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4473745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4474308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4474819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4475387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4475821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4476249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:21:38.4476717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:38.4477155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:21:38.4477609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:38.4477951Z ok (3.913s)
2022-11-23T02:21:38.4478098Z 
2022-11-23T02:21:38.4478365Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4478669Z Ran 1 test in 3.913s
2022-11-23T02:21:38.4478830Z 
2022-11-23T02:21:38.4478925Z OK
2022-11-23T02:21:38.4479057Z 
2022-11-23T02:21:38.4479181Z Generating XML reports...
2022-11-23T02:21:38.4479795Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123022113.xml
2022-11-23T02:21:38.4480524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4480965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4481526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4482047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4482271Z 
2022-11-23T02:21:38.4482379Z Running tests...
2022-11-23T02:21:38.4482778Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4483290Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4483796Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4484286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53675
2022-11-23T02:21:38.4484728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53676
2022-11-23T02:21:38.4485146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53677
2022-11-23T02:21:38.4485575Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53678
2022-11-23T02:21:38.4486172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4486600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4487166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4487630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4488197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4488616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4489178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4489631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4490219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4490662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4491221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4491729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4492285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4492726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4493287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4493738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4494146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:21:38.4494768Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T02:21:38.4495265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:38.4495866Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T02:21:38.4496355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:21:38.4496971Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T02:21:38.4497462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:38.4498441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:21:38.4499042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:38.4499626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:21:38.4500105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:38.4500754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:21:38.4501432Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:21:38.4502107Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:21:38.4502779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:21:38.4503141Z ok (6.759s)
2022-11-23T02:21:38.4503287Z 
2022-11-23T02:21:38.4503551Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4503875Z Ran 1 test in 6.759s
2022-11-23T02:21:38.4504035Z 
2022-11-23T02:21:38.4504110Z OK
2022-11-23T02:21:38.4504239Z 
2022-11-23T02:21:38.4504363Z Generating XML reports...
2022-11-23T02:21:38.4505000Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123022119.xml
2022-11-23T02:21:38.4505736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4506162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4506726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4507190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4507414Z 
2022-11-23T02:21:38.4507505Z Running tests...
2022-11-23T02:21:38.4507900Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4508428Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4508911Z   test_op_isinstance_of_reduceop (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4509316Z ok (1.589s)
2022-11-23T02:21:38.4509475Z 
2022-11-23T02:21:38.4509735Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4510051Z Ran 1 test in 1.589s
2022-11-23T02:21:38.4510211Z 
2022-11-23T02:21:38.4510286Z OK
2022-11-23T02:21:38.4510417Z 
2022-11-23T02:21:38.4510540Z Generating XML reports...
2022-11-23T02:21:38.4511089Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123022128.xml
2022-11-23T02:21:38.4511757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4512182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4512748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4513209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4513434Z 
2022-11-23T02:21:38.4513529Z Running tests...
2022-11-23T02:21:38.4513927Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4514450Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4514920Z   test_reduceop_copyable (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4515242Z ok (1.649s)
2022-11-23T02:21:38.4515385Z 
2022-11-23T02:21:38.4515648Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4515968Z Ran 1 test in 1.649s
2022-11-23T02:21:38.4516124Z 
2022-11-23T02:21:38.4516200Z OK
2022-11-23T02:21:38.4516331Z 
2022-11-23T02:21:38.4516516Z Generating XML reports...
2022-11-23T02:21:38.4517064Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123022132.xml
2022-11-23T02:21:38.4517735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:38.4518159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:38.4518726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:38.4519185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:38.4519409Z 
2022-11-23T02:21:38.4519500Z Running tests...
2022-11-23T02:21:38.4519894Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4520416Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T02:21:38.4520886Z   test_reduceop_pickle (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:38.4521213Z ok (1.638s)
2022-11-23T02:21:38.4521358Z 
2022-11-23T02:21:38.4521617Z ----------------------------------------------------------------------
2022-11-23T02:21:38.4521935Z Ran 1 test in 1.638s
2022-11-23T02:21:38.4522093Z 
2022-11-23T02:21:38.4522172Z OK
2022-11-23T02:21:38.4522302Z 
2022-11-23T02:21:38.4522425Z Generating XML reports...
2022-11-23T02:21:38.4522970Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123022136.xml
2022-11-23T02:21:38.4523292Z 
2022-11-23T02:21:38.4523665Z ##[endgroup]
2022-11-23T02:21:38.4524227Z FINISHED PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_voatt834)
2022-11-23T02:21:38.4524554Z 
2022-11-23T02:21:38.4524844Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-11-23 02:21:38.436432]
2022-11-23T02:21:38.4525549Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:38.436703]
2022-11-23T02:22:19.0330126Z 
2022-11-23T02:22:19.0331081Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-11-23T02:22:19.0332675Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_bdlnidmq)
2022-11-23T02:22:19.0333685Z 
2022-11-23T02:22:19.0335744Z Running tests...
2022-11-23T02:22:19.0336265Z ----------------------------------------------------------------------
2022-11-23T02:22:19.0337053Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-11-23T02:22:19.0337623Z   test_grad_scaling (__main__.TestShardGradScaler) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:22:19.0338343Z ok (1.664s)
2022-11-23T02:22:19.0341908Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... ok (0.002s)
2022-11-23T02:22:19.0342378Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... ok (0.006s)
2022-11-23T02:22:19.0343046Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53972
2022-11-23T02:22:19.0343675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53973
2022-11-23T02:22:19.0344352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0344789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0345367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0345838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0346623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0347055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0347634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0348101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0348625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:19.0349116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:19.0349759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0350454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0350985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:19.0351453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:19.0351917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0352399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0353676Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0354524Z   warnings.warn(
2022-11-23T02:22:19.0355766Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0356549Z   warnings.warn(
2022-11-23T02:22:19.0356901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0357381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0357740Z dist init r=0, world=2
2022-11-23T02:22:19.0357992Z dist init r=1, world=2
2022-11-23T02:22:19.0358210Z ok (4.514s)
2022-11-23T02:22:19.0358728Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54055
2022-11-23T02:22:19.0359338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54056
2022-11-23T02:22:19.0359935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0360376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0360948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0361415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0361969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0362402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0362971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0363483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0363929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:19.0364424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:19.0365078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0365744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0366259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:19.0366724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:19.0367201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0367660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0368926Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0369697Z   warnings.warn(
2022-11-23T02:22:19.0370837Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0371607Z   warnings.warn(
2022-11-23T02:22:19.0372013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0372499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0372857Z dist init r=0, world=2
2022-11-23T02:22:19.0373089Z dist init r=1, world=2
2022-11-23T02:22:19.0373326Z ok (4.512s)
2022-11-23T02:22:19.0373867Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54138
2022-11-23T02:22:19.0374494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54139
2022-11-23T02:22:19.0375085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0375531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0376103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0376552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0377126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0377562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0378679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0379520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0380353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:19.0381398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:19.0382368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0383037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0383554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:19.0384019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:19.0384470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0384946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0386216Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0386992Z   warnings.warn(
2022-11-23T02:22:19.0388125Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0388888Z   warnings.warn(
2022-11-23T02:22:19.0389242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0389720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0390120Z dist init r=1, world=2
2022-11-23T02:22:19.0390351Z dist init r=0, world=2
2022-11-23T02:22:19.0390667Z ok (4.512s)
2022-11-23T02:22:19.0391209Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54221
2022-11-23T02:22:19.0391807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54222
2022-11-23T02:22:19.0392418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0392860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0393432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0393885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0394454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0394890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0395456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0395898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0396341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:19.0396830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:19.0397458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0398209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0398723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:19.0399190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:19.0399645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0400122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0401384Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0402158Z   warnings.warn(
2022-11-23T02:22:19.0403308Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0404051Z   warnings.warn(
2022-11-23T02:22:19.0404420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0404899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0405256Z dist init r=1, world=2
2022-11-23T02:22:19.0405491Z dist init r=0, world=2
2022-11-23T02:22:19.0405727Z ok (4.612s)
2022-11-23T02:22:19.0406259Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54304
2022-11-23T02:22:19.0406920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54305
2022-11-23T02:22:19.0407535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0407978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0408546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0408996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0409565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0410006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0410552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0411013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0411458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:19.0411945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:19.0412576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0413253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0413772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:19.0414306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:19.0414760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0415243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0416511Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0417282Z   warnings.warn(
2022-11-23T02:22:19.0418902Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0419694Z   warnings.warn(
2022-11-23T02:22:19.0419961Z   File "<string>", line 1, in <module>
2022-11-23T02:22:19.0420327Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:22:19.0420679Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:22:19.0421046Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:22:19.0421411Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:22:19.0421778Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:22:19.0422112Z     self.run()
2022-11-23T02:22:19.0422445Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:22:19.0422807Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:22:19.0423493Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:22:19.0423971Z     self.run_test(test_name, pipe)
2022-11-23T02:22:19.0424506Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:22:19.0424875Z     getattr(self, test_name)()
2022-11-23T02:22:19.0425388Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:22:19.0425751Z     fn()
2022-11-23T02:22:19.0426241Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:22:19.0426615Z     test(self, **param_kwargs)
2022-11-23T02:22:19.0427117Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:22:19.0427508Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0427929Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:22:19.0428343Z     self._test_fsdp_parity(
2022-11-23T02:22:19.0428857Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:22:19.0429271Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:22:19.0429800Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:22:19.0430189Z     output = model(*input)
2022-11-23T02:22:19.0430659Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:22:19.0431023Z     return forward_call(*input, **kwargs)
2022-11-23T02:22:19.0431643Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:22:19.0432093Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:22:19.0432658Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:22:19.0433028Z     _lazy_init(state, module)
2022-11-23T02:22:19.0433523Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:22:19.0433923Z     handle.init_flat_param_attributes()
2022-11-23T02:22:19.0434416Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:22:19.0434795Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0435323Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:22:19.0435704Z     p_assert(
2022-11-23T02:22:19.0436150Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:22:19.0436527Z     traceback.print_stack()
2022-11-23T02:22:19.0436810Z   File "<string>", line 1, in <module>
2022-11-23T02:22:19.0437158Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:22:19.0437521Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:22:19.0437888Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:22:19.0453457Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:22:19.0453882Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:22:19.0454233Z     self.run()
2022-11-23T02:22:19.0454566Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:22:19.0454915Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:22:19.0455467Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:22:19.0455855Z     self.run_test(test_name, pipe)
2022-11-23T02:22:19.0456468Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:22:19.0456882Z     getattr(self, test_name)()
2022-11-23T02:22:19.0457397Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:22:19.0457765Z     fn()
2022-11-23T02:22:19.0458755Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:22:19.0459155Z     test(self, **param_kwargs)
2022-11-23T02:22:19.0459667Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:22:19.0460039Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0460481Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:22:19.0460894Z     self._test_fsdp_parity(
2022-11-23T02:22:19.0461415Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:22:19.0461813Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:22:19.0462361Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:22:19.0462755Z     output = model(*input)
2022-11-23T02:22:19.0463211Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:22:19.0463594Z     return forward_call(*input, **kwargs)
2022-11-23T02:22:19.0464140Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:22:19.0464717Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:22:19.0465267Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:22:19.0465664Z     _lazy_init(state, module)
2022-11-23T02:22:19.0466170Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:22:19.0466557Z     handle.init_flat_param_attributes()
2022-11-23T02:22:19.0467064Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:22:19.0467438Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0467948Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:22:19.0468329Z     p_assert(
2022-11-23T02:22:19.0468797Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:22:19.0469184Z     traceback.print_stack()
2022-11-23T02:22:19.0469557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0470040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0470523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0470979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0471337Z dist init r=0, world=2
2022-11-23T02:22:19.0471804Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T02:22:19.0472249Z dist init r=1, world=2
2022-11-23T02:22:19.0472691Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T02:22:19.0473119Z ok (4.612s)
2022-11-23T02:22:19.0473638Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54387
2022-11-23T02:22:19.0474317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54388
2022-11-23T02:22:19.0474919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0475369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0475941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0476389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0476963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0477411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0477979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0478423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0478870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:19.0479363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:19.0479997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0480681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0481197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:19.0481726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:19.0482184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0482665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0483931Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0484715Z   warnings.warn(
2022-11-23T02:22:19.0485857Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0486605Z   warnings.warn(
2022-11-23T02:22:19.0486874Z   File "<string>", line 1, in <module>
2022-11-23T02:22:19.0487246Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:22:19.0487618Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:22:19.0487964Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:22:19.0488330Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:22:19.0488718Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:22:19.0489034Z     self.run()
2022-11-23T02:22:19.0489359Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:22:19.0489722Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:22:19.0490260Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:22:19.0490707Z     self.run_test(test_name, pipe)
2022-11-23T02:22:19.0491245Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:22:19.0491636Z     getattr(self, test_name)()
2022-11-23T02:22:19.0492129Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:22:19.0492497Z     fn()
2022-11-23T02:22:19.0492987Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:22:19.0493360Z     test(self, **param_kwargs)
2022-11-23T02:22:19.0493870Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:22:19.0494261Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0494695Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:22:19.0495095Z     self._test_fsdp_parity(
2022-11-23T02:22:19.0495613Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:22:19.0496032Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:22:19.0496558Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:22:19.0496949Z     output = model(*input)
2022-11-23T02:22:19.0497422Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:22:19.0497810Z     return forward_call(*input, **kwargs)
2022-11-23T02:22:19.0498883Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:22:19.0499338Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:22:19.0499900Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:22:19.0500272Z     _lazy_init(state, module)
2022-11-23T02:22:19.0500768Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:22:19.0501171Z     handle.init_flat_param_attributes()
2022-11-23T02:22:19.0501678Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:22:19.0502036Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0502565Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:22:19.0502951Z     p_assert(
2022-11-23T02:22:19.0503398Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:22:19.0503771Z     traceback.print_stack()
2022-11-23T02:22:19.0504055Z   File "<string>", line 1, in <module>
2022-11-23T02:22:19.0504400Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:22:19.0504761Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:22:19.0505128Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:22:19.0505494Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:22:19.0505858Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:22:19.0506188Z     self.run()
2022-11-23T02:22:19.0506513Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:22:19.0506855Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:22:19.0507360Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:22:19.0507748Z     self.run_test(test_name, pipe)
2022-11-23T02:22:19.0508243Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:22:19.0508822Z     getattr(self, test_name)()
2022-11-23T02:22:19.0509348Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:22:19.0509710Z     fn()
2022-11-23T02:22:19.0510181Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:22:19.0510572Z     test(self, **param_kwargs)
2022-11-23T02:22:19.0511079Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:22:19.0511448Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0511887Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:22:19.0512297Z     self._test_fsdp_parity(
2022-11-23T02:22:19.0512812Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:22:19.0513211Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:22:19.0513759Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:22:19.0514151Z     output = model(*input)
2022-11-23T02:22:19.0514603Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:22:19.0514980Z     return forward_call(*input, **kwargs)
2022-11-23T02:22:19.0515520Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:22:19.0515973Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:22:19.0516605Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:22:19.0516997Z     _lazy_init(state, module)
2022-11-23T02:22:19.0517500Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:22:19.0517878Z     handle.init_flat_param_attributes()
2022-11-23T02:22:19.0518386Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:22:19.0518767Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0519277Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:22:19.0519655Z     p_assert(
2022-11-23T02:22:19.0520117Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:22:19.0520502Z     traceback.print_stack()
2022-11-23T02:22:19.0520874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0521356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0521835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0522290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0522650Z dist init r=0, world=2
2022-11-23T02:22:19.0523115Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T02:22:19.0523557Z dist init r=1, world=2
2022-11-23T02:22:19.0523997Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T02:22:19.0524418Z ok (4.612s)
2022-11-23T02:22:19.0524966Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54470
2022-11-23T02:22:19.0525650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54471
2022-11-23T02:22:19.0526248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0526694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0527267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0527716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0528290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0528734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0529300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0529742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0530193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:19.0530688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:19.0531323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0532001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0532518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:19.0533047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:19.0533505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0533987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0535252Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0536028Z   warnings.warn(
2022-11-23T02:22:19.0537165Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0538241Z   warnings.warn(
2022-11-23T02:22:19.0538603Z   File "<string>", line 1, in <module>
2022-11-23T02:22:19.0538977Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:22:19.0539346Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:22:19.0539697Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:22:19.0540060Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:22:19.0540445Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:22:19.0540759Z     self.run()
2022-11-23T02:22:19.0541085Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:22:19.0541451Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:22:19.0541954Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:22:19.0542431Z     self.run_test(test_name, pipe)
2022-11-23T02:22:19.0542968Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:22:19.0543355Z     getattr(self, test_name)()
2022-11-23T02:22:19.0543845Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:22:19.0544210Z     fn()
2022-11-23T02:22:19.0544700Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:22:19.0545075Z     test(self, **param_kwargs)
2022-11-23T02:22:19.0545585Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:22:19.0545977Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0546413Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:22:19.0546809Z     self._test_fsdp_parity(
2022-11-23T02:22:19.0547324Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:22:19.0547741Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:22:19.0548270Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:22:19.0548661Z     output = model(*input)
2022-11-23T02:22:19.0549132Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:22:19.0549521Z     return forward_call(*input, **kwargs)
2022-11-23T02:22:19.0550043Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:22:19.0550574Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:22:19.0551141Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:22:19.0551511Z     _lazy_init(state, module)
2022-11-23T02:22:19.0552002Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:22:19.0552405Z     handle.init_flat_param_attributes()
2022-11-23T02:22:19.0552912Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:22:19.0553268Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0553803Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:22:19.0554184Z     p_assert(
2022-11-23T02:22:19.0554637Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:22:19.0555010Z     traceback.print_stack()
2022-11-23T02:22:19.0555293Z   File "<string>", line 1, in <module>
2022-11-23T02:22:19.0555645Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:22:19.0556009Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:22:19.0556375Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:22:19.0556742Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:22:19.0557105Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:22:19.0557437Z     self.run()
2022-11-23T02:22:19.0557766Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:22:19.0558108Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:22:19.0558615Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:22:19.0559005Z     self.run_test(test_name, pipe)
2022-11-23T02:22:19.0559507Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:22:19.0559946Z     getattr(self, test_name)()
2022-11-23T02:22:19.0560468Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:22:19.0560834Z     fn()
2022-11-23T02:22:19.0561297Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:22:19.0561682Z     test(self, **param_kwargs)
2022-11-23T02:22:19.0562194Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:22:19.0562562Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0562998Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:22:19.0563416Z     self._test_fsdp_parity(
2022-11-23T02:22:19.0563932Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:22:19.0564335Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:22:19.0564886Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:22:19.0565279Z     output = model(*input)
2022-11-23T02:22:19.0565729Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:22:19.0566113Z     return forward_call(*input, **kwargs)
2022-11-23T02:22:19.0566655Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:22:19.0567105Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:22:19.0567726Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:22:19.0568120Z     _lazy_init(state, module)
2022-11-23T02:22:19.0568622Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:22:19.0569009Z     handle.init_flat_param_attributes()
2022-11-23T02:22:19.0569517Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:22:19.0569894Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0570422Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:22:19.0570781Z     p_assert(
2022-11-23T02:22:19.0571244Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:22:19.0571619Z     traceback.print_stack()
2022-11-23T02:22:19.0571994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0572476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0572957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0573412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0573772Z dist init r=1, world=2
2022-11-23T02:22:19.0574236Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T02:22:19.0574677Z dist init r=0, world=2
2022-11-23T02:22:19.0575119Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T02:22:19.0575540Z ok (4.612s)
2022-11-23T02:22:19.0576071Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54553
2022-11-23T02:22:19.0576741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54554
2022-11-23T02:22:19.0577342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0577787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0579043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0579500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0580079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:19.0580519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:19.0581093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:19.0581534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:19.0581989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:19.0582483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:19.0583129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0583791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:19.0584307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:19.0584776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:19.0585333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0585810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0587081Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0587858Z   warnings.warn(
2022-11-23T02:22:19.0588998Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:19.0589756Z   warnings.warn(
2022-11-23T02:22:19.0590045Z   File "<string>", line 1, in <module>
2022-11-23T02:22:19.0590419Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:22:19.0590788Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:22:19.0591139Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:22:19.0591504Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:22:19.0591890Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:22:19.0592205Z     self.run()
2022-11-23T02:22:19.0592534Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:22:19.0592900Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:22:19.0593394Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:22:19.0593783Z     self.run_test(test_name, pipe)
2022-11-23T02:22:19.0594385Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:22:19.0594789Z     getattr(self, test_name)()
2022-11-23T02:22:19.0595292Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:22:19.0595664Z     fn()
2022-11-23T02:22:19.0596158Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:22:19.0596535Z     test(self, **param_kwargs)
2022-11-23T02:22:19.0597036Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:22:19.0597423Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0597861Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:22:19.0598256Z     self._test_fsdp_parity(
2022-11-23T02:22:19.0598778Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:22:19.0599206Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:22:19.0599742Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:22:19.0600136Z     output = model(*input)
2022-11-23T02:22:19.0600613Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:22:19.0601003Z     return forward_call(*input, **kwargs)
2022-11-23T02:22:19.0601531Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:22:19.0602074Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:22:19.0602641Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:22:19.0603032Z     _lazy_init(state, module)
2022-11-23T02:22:19.0603511Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:22:19.0603913Z     handle.init_flat_param_attributes()
2022-11-23T02:22:19.0604420Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:22:19.0604775Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0605339Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:22:19.0605723Z     p_assert(
2022-11-23T02:22:19.0606183Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:22:19.0606566Z     traceback.print_stack()
2022-11-23T02:22:19.0606849Z   File "<string>", line 1, in <module>
2022-11-23T02:22:19.0607219Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:22:19.0607566Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:22:19.0607933Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:22:19.0608301Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:22:19.0608671Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:22:19.0609015Z     self.run()
2022-11-23T02:22:19.0609366Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:22:19.0609738Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:22:19.0610250Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:22:19.0610617Z     self.run_test(test_name, pipe)
2022-11-23T02:22:19.0611136Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:22:19.0611581Z     getattr(self, test_name)()
2022-11-23T02:22:19.0612082Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:22:19.0612441Z     fn()
2022-11-23T02:22:19.0612923Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:22:19.0613295Z     test(self, **param_kwargs)
2022-11-23T02:22:19.0613796Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:22:19.0614184Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0614615Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:22:19.0615011Z     self._test_fsdp_parity(
2022-11-23T02:22:19.0615528Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:22:19.0615942Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:22:19.0616473Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:22:19.0616857Z     output = model(*input)
2022-11-23T02:22:19.0617328Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:22:19.0617703Z     return forward_call(*input, **kwargs)
2022-11-23T02:22:19.0618633Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:22:19.0619112Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:22:19.0619783Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:22:19.0620157Z     _lazy_init(state, module)
2022-11-23T02:22:19.0620663Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:22:19.0621076Z     handle.init_flat_param_attributes()
2022-11-23T02:22:19.0621597Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:22:19.0621990Z     return func(*args, **kwargs)
2022-11-23T02:22:19.0622500Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:22:19.0622877Z     p_assert(
2022-11-23T02:22:19.0623339Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:22:19.0623696Z     traceback.print_stack()
2022-11-23T02:22:19.0624089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0624567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0625046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0625500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:19.0625856Z dist init r=0, world=2
2022-11-23T02:22:19.0626322Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T02:22:19.0626744Z dist init r=1, world=2
2022-11-23T02:22:19.0627202Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T02:22:19.0627627Z ok (4.612s)
2022-11-23T02:22:19.0627776Z 
2022-11-23T02:22:19.0628055Z ----------------------------------------------------------------------
2022-11-23T02:22:19.0628370Z Ran 11 tests in 38.269s
2022-11-23T02:22:19.0628535Z 
2022-11-23T02:22:19.0628629Z OK
2022-11-23T02:22:19.0628763Z 
2022-11-23T02:22:19.0628886Z Generating XML reports...
2022-11-23T02:22:19.0629574Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123022140.xml
2022-11-23T02:22:19.0630500Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123022140.xml
2022-11-23T02:22:19.0630944Z 
2022-11-23T02:22:19.0631338Z ##[endgroup]
2022-11-23T02:22:19.0631987Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_bdlnidmq)
2022-11-23T02:22:19.0632373Z 
2022-11-23T02:22:19.0632616Z Running distributed/fsdp/test_fsdp_comm ... [2022-11-23 02:22:19.033502]
2022-11-23T02:22:19.0633287Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:19.033770]
2022-11-23T02:23:00.3210360Z 
2022-11-23T02:23:00.3210845Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm
2022-11-23T02:23:00.3212232Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_xwja7sx2)
2022-11-23T02:23:00.3215551Z 
2022-11-23T02:23:00.3215959Z Running tests...
2022-11-23T02:23:00.3216529Z ----------------------------------------------------------------------
2022-11-23T02:23:00.3217077Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm
2022-11-23T02:23:00.3217603Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T02:23:00.3218550Z Tests FSDP's communication cost in terms of calls to collective ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:23:00.3219321Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54671
2022-11-23T02:23:00.3219788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54672
2022-11-23T02:23:00.3222198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3223076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3223729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3224208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3224788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3225216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3225803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3226265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3226703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:00.3227200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:00.3227858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3228549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3229052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:00.3229523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:00.3229883Z dist init r=1, world=2
2022-11-23T02:23:00.3230120Z dist init r=0, world=2
2022-11-23T02:23:00.3230394Z ok (6.461s)
2022-11-23T02:23:00.3231413Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T02:23:00.3232625Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54754
2022-11-23T02:23:00.3233837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54755
2022-11-23T02:23:00.3234792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3235242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3235821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3236281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3236862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3237390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3237967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3238435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3238867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:00.3239357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:00.3240009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3240790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3241291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:00.3241760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:00.3242120Z dist init r=1, world=2
2022-11-23T02:23:00.3242356Z dist init r=0, world=2
2022-11-23T02:23:00.3242596Z ok (4.713s)
2022-11-23T02:23:00.3242976Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T02:23:00.3243676Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54837
2022-11-23T02:23:00.3244187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54838
2022-11-23T02:23:00.3244790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3245244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3245799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3246268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3246844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3247288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3247837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3248301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3248752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:00.3249247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:00.3249882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3250623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3251152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:00.3251603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:00.3251959Z dist init r=1, world=2
2022-11-23T02:23:00.3252210Z dist init r=0, world=2
2022-11-23T02:23:00.3252452Z ok (4.914s)
2022-11-23T02:23:00.3252855Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T02:23:00.3253590Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54920
2022-11-23T02:23:00.3254130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54921
2022-11-23T02:23:00.3254723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3255170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3255739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3256204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3256759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3257202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3257772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3258777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3259297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:00.3259925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:00.3260588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3261252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3261772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:00.3262240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:00.3262604Z dist init r=1, world=2
2022-11-23T02:23:00.3262839Z dist init r=0, world=2
2022-11-23T02:23:00.3263079Z ok (4.913s)
2022-11-23T02:23:00.3263459Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T02:23:00.3264141Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55003
2022-11-23T02:23:00.3264669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55004
2022-11-23T02:23:00.3265269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3265720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3266273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3266739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3267312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3267733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3268398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3268870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3269320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:00.3269793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:00.3270439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3271121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3271643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:00.3272089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:00.3273338Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:00.3274163Z   warnings.warn(
2022-11-23T02:23:00.3275315Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:00.3276173Z   warnings.warn(
2022-11-23T02:23:00.3276407Z dist init r=0, world=2
2022-11-23T02:23:00.3276662Z dist init r=1, world=2
2022-11-23T02:23:00.3276901Z ok (4.413s)
2022-11-23T02:23:00.3277321Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T02:23:00.3278039Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55086
2022-11-23T02:23:00.3278568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55087
2022-11-23T02:23:00.3279169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3279605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3280175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3280646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3281221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3281645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3282216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3282677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3283124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:00.3283598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:00.3284252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3284987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3285494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:00.3285959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:00.3287206Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:00.3287982Z   warnings.warn(
2022-11-23T02:23:00.3289125Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:00.3289925Z   warnings.warn(
2022-11-23T02:23:00.3290159Z dist init r=0, world=2
2022-11-23T02:23:00.3290412Z dist init r=1, world=2
2022-11-23T02:23:00.3290657Z ok (4.513s)
2022-11-23T02:23:00.3291015Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T02:23:00.3291770Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55169
2022-11-23T02:23:00.3292304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55170
2022-11-23T02:23:00.3292891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3293340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3293912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3294378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3294931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3295375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3295944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3296411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3296842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:00.3297336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:00.3298430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3299117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3299633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:00.3300099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:00.3301443Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:00.3302229Z   warnings.warn(
2022-11-23T02:23:00.3303351Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:00.3304129Z   warnings.warn(
2022-11-23T02:23:00.3304383Z dist init r=0, world=2
2022-11-23T02:23:00.3304637Z dist init r=1, world=2
2022-11-23T02:23:00.3304859Z ok (4.513s)
2022-11-23T02:23:00.3305281Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T02:23:00.3306009Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55252
2022-11-23T02:23:00.3306522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55253
2022-11-23T02:23:00.3307122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3307566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3308141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3308663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3309234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:00.3309678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:00.3310244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:00.3310686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:00.3311135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:00.3311624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:00.3312256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3312946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:00.3313464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:00.3313937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:00.3315168Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:00.3315941Z   warnings.warn(
2022-11-23T02:23:00.3317147Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:00.3317927Z   warnings.warn(
2022-11-23T02:23:00.3318180Z dist init r=0, world=2
2022-11-23T02:23:00.3318413Z dist init r=1, world=2
2022-11-23T02:23:00.3318652Z ok (4.512s)
2022-11-23T02:23:00.3318802Z 
2022-11-23T02:23:00.3319074Z ----------------------------------------------------------------------
2022-11-23T02:23:00.3319387Z Ran 8 tests in 38.953s
2022-11-23T02:23:00.3319547Z 
2022-11-23T02:23:00.3319642Z OK
2022-11-23T02:23:00.3319777Z 
2022-11-23T02:23:00.3319906Z Generating XML reports...
2022-11-23T02:23:00.3320495Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221123022220.xml
2022-11-23T02:23:00.3320826Z 
2022-11-23T02:23:00.3321230Z ##[endgroup]
2022-11-23T02:23:00.3321816Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_xwja7sx2)
2022-11-23T02:23:00.3322162Z 
2022-11-23T02:23:00.3322456Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-11-23 02:23:00.321196]
2022-11-23T02:23:00.3323142Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:23:00.321462]
2022-11-23T02:23:47.3190582Z 
2022-11-23T02:23:47.3191333Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_freezing_weights
2022-11-23T02:23:47.3192307Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_31hljhwq)
2022-11-23T02:23:47.3192947Z 
2022-11-23T02:23:47.3194573Z Running tests...
2022-11-23T02:23:47.3195119Z ----------------------------------------------------------------------
2022-11-23T02:23:47.3195736Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights
2022-11-23T02:23:47.3196415Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:23:47.3197258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55370
2022-11-23T02:23:47.3197697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55371
2022-11-23T02:23:47.3198334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3198789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3199354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3201009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3201706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3202202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3202796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3203271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3203705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:47.3204208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:47.3204877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3205568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3206098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:47.3206755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:47.3207261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3207725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3208092Z dist init r=0, world=2
2022-11-23T02:23:47.3208350Z dist init r=1, world=2
2022-11-23T02:23:47.3208572Z ok (7.072s)
2022-11-23T02:23:47.3209233Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55453
2022-11-23T02:23:47.3209962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55454
2022-11-23T02:23:47.3210570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3211023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3211601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3212072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3212631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3213076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3213646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3214181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3214638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:47.3215134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:47.3215792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3216457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3216978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:47.3217446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:47.3218341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3218831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3220115Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:47.3220889Z   warnings.warn(
2022-11-23T02:23:47.3222045Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:47.3222819Z   warnings.warn(
2022-11-23T02:23:47.3223052Z dist init r=0, world=2
2022-11-23T02:23:47.3223308Z dist init r=1, world=2
2022-11-23T02:23:47.3223653Z ok (5.414s)
2022-11-23T02:23:47.3224214Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55536
2022-11-23T02:23:47.3224865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55537
2022-11-23T02:23:47.3225477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3225926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3226479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3226952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3227529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3227977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3228529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3228991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3229440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:47.3229912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:47.3230566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3231347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3231869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:47.3232317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:47.3232785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3233270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3233613Z dist init r=1, world=2
2022-11-23T02:23:47.3233867Z dist init r=0, world=2
2022-11-23T02:23:47.3234108Z ok (5.313s)
2022-11-23T02:23:47.3234667Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55619
2022-11-23T02:23:47.3235303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55620
2022-11-23T02:23:47.3235914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3236369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3236927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3237404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3237984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3238431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3238976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3239445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3239894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:47.3240449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:47.3241098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3241789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3242308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:47.3242758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:47.3243232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3243715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3244984Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:47.3245754Z   warnings.warn(
2022-11-23T02:23:47.3246890Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:47.3247727Z   warnings.warn(
2022-11-23T02:23:47.3247981Z dist init r=0, world=2
2022-11-23T02:23:47.3248237Z dist init r=1, world=2
2022-11-23T02:23:47.3248469Z ok (5.313s)
2022-11-23T02:23:47.3249030Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55702
2022-11-23T02:23:47.3249677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55703
2022-11-23T02:23:47.3250292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3250721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3251293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3251766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3252323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3252772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3253350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3253814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3254244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:47.3254733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:47.3255387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3256064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3256583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:47.3257108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:47.3257592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3258313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3258686Z dist init r=0, world=2
2022-11-23T02:23:47.3258942Z dist init r=1, world=2
2022-11-23T02:23:47.3259164Z ok (5.413s)
2022-11-23T02:23:47.3259720Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55785
2022-11-23T02:23:47.3260373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55786
2022-11-23T02:23:47.3260988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3261425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3262000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3262468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3263044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3263466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3264034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3264602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3265031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:47.3265527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:47.3266183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3266873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3267373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:47.3267845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:47.3268315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3268801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3270052Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:47.3270831Z   warnings.warn(
2022-11-23T02:23:47.3271978Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:47.3272754Z   warnings.warn(
2022-11-23T02:23:47.3273007Z dist init r=1, world=2
2022-11-23T02:23:47.3273237Z dist init r=0, world=2
2022-11-23T02:23:47.3273478Z ok (5.513s)
2022-11-23T02:23:47.3274113Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55898
2022-11-23T02:23:47.3274749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55899
2022-11-23T02:23:47.3275363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3275817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3276368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3276836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3277401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3277838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3278391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3278848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3279292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:47.3279776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:47.3280408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3281155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3281670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:47.3282121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:47.3282593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3283069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3283425Z dist init r=1, world=2
2022-11-23T02:23:47.3283655Z dist init r=0, world=2
2022-11-23T02:23:47.3283891Z ok (5.313s)
2022-11-23T02:23:47.3284445Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55981
2022-11-23T02:23:47.3285077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55982
2022-11-23T02:23:47.3285683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3286130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3286698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3287146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3287717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:47.3288154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:47.3288720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:47.3289169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:47.3289651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:47.3290199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:47.3290842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3291519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:47.3292031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:47.3292492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:47.3292946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3293425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:23:47.3294691Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:47.3295460Z   warnings.warn(
2022-11-23T02:23:47.3296609Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:47.3297435Z   warnings.warn(
2022-11-23T02:23:47.3297682Z dist init r=1, world=2
2022-11-23T02:23:47.3298236Z dist init r=0, world=2
2022-11-23T02:23:47.3298469Z ok (5.313s)
2022-11-23T02:23:47.3298622Z 
2022-11-23T02:23:47.3298899Z ----------------------------------------------------------------------
2022-11-23T02:23:47.3299226Z Ran 8 tests in 44.666s
2022-11-23T02:23:47.3299387Z 
2022-11-23T02:23:47.3299479Z OK
2022-11-23T02:23:47.3299593Z 
2022-11-23T02:23:47.3299718Z Generating XML reports...
2022-11-23T02:23:47.3300341Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221123022302.xml
2022-11-23T02:23:47.3300705Z 
2022-11-23T02:23:47.3301123Z ##[endgroup]
2022-11-23T02:23:47.3301743Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_31hljhwq)
2022-11-23T02:23:47.3302128Z 
2022-11-23T02:23:47.3302368Z Running distributed/test_store ... [2022-11-23 02:23:47.319302]
2022-11-23T02:23:47.3303041Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:23:47.319672]
2022-11-23T02:25:53.3220508Z 
2022-11-23T02:25:53.3221510Z Expand the folded group to see the log file of distributed/test_store
2022-11-23T02:25:53.3222380Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_4zf70t7m)
2022-11-23T02:25:53.3223133Z <unittest.suite.TestSuite tests=[<__main__.FileStoreTest testMethod=test_compare_set>, <__main__.FileStoreTest testMethod=test_init_pg_and_rpc_with_same_file>, <__main__.FileStoreTest testMethod=test_refcount>, <__main__.FileStoreTest testMethod=test_set_get>]>
2022-11-23T02:25:53.3223717Z test_compare_set (__main__.FileStoreTest)
2022-11-23T02:25:53.3224091Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest)
2022-11-23T02:25:53.3224426Z test_refcount (__main__.FileStoreTest)
2022-11-23T02:25:53.3224724Z test_set_get (__main__.FileStoreTest)
2022-11-23T02:25:53.3225426Z <unittest.suite.TestSuite tests=[<__main__.HashStoreTest testMethod=test_compare_set>, <__main__.HashStoreTest testMethod=test_set_get>]>
2022-11-23T02:25:53.3225891Z test_compare_set (__main__.HashStoreTest)
2022-11-23T02:25:53.3226174Z test_set_get (__main__.HashStoreTest)
2022-11-23T02:25:53.3230185Z <unittest.suite.TestSuite tests=[<__main__.PrefixFileStoreTest testMethod=test_compare_set>, <__main__.PrefixFileStoreTest testMethod=test_set_get>]>
2022-11-23T02:25:53.3230745Z test_compare_set (__main__.PrefixFileStoreTest)
2022-11-23T02:25:53.3231092Z test_set_get (__main__.PrefixFileStoreTest)
2022-11-23T02:25:53.3231519Z <unittest.suite.TestSuite tests=[<__main__.PrefixStoreTest testMethod=test_get_underlying_store>]>
2022-11-23T02:25:53.3231942Z test_get_underlying_store (__main__.PrefixStoreTest)
2022-11-23T02:25:53.3232452Z <unittest.suite.TestSuite tests=[<__main__.PrefixTCPStoreTest testMethod=test_compare_set>, <__main__.PrefixTCPStoreTest testMethod=test_set_get>]>
2022-11-23T02:25:53.3233094Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-11-23T02:25:53.3233446Z test_set_get (__main__.PrefixTCPStoreTest)
2022-11-23T02:25:53.3233822Z <unittest.suite.TestSuite tests=[<__main__.PythonStoreTest testMethod=test_set_get>]>
2022-11-23T02:25:53.3234209Z test_set_get (__main__.PythonStoreTest)
2022-11-23T02:25:53.3237422Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_nominal>]>
2022-11-23T02:25:53.3237814Z test_nominal (__main__.RendezvousEnvTest)
2022-11-23T02:25:53.3238318Z <unittest.suite.TestSuite tests=[<__main__.RendezvousFileTest testMethod=test_common_errors>, <__main__.RendezvousFileTest testMethod=test_nominal>]>
2022-11-23T02:25:53.3238804Z test_common_errors (__main__.RendezvousFileTest)
2022-11-23T02:25:53.3239303Z test_nominal (__main__.RendezvousFileTest)
2022-11-23T02:25:53.3240044Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTCPTest testMethod=test_common_errors>, <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]>
2022-11-23T02:25:53.3241158Z test_common_errors (__main__.RendezvousTCPTest)
2022-11-23T02:25:53.3241855Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-11-23T02:25:53.3242578Z test_nominal (__main__.RendezvousTCPTest)
2022-11-23T02:25:53.3243111Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-11-23T02:25:53.3243626Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTest testMethod=test_unknown_handler>, <__main__.RendezvousTest testMethod=test_url_with_node_params>]>
2022-11-23T02:25:53.3244111Z test_unknown_handler (__main__.RendezvousTest)
2022-11-23T02:25:53.3245315Z test_url_with_node_params (__main__.RendezvousTest)
2022-11-23T02:25:53.3246875Z <unittest.suite.TestSuite tests=[<__main__.TCPStoreTest testMethod=test_address_already_in_use>, <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]>
2022-11-23T02:25:53.3248829Z test_address_already_in_use (__main__.TCPStoreTest)
2022-11-23T02:25:53.3249300Z test_compare_set (__main__.TCPStoreTest)
2022-11-23T02:25:53.3249862Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-11-23T02:25:53.3250391Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-11-23T02:25:53.3250757Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-11-23T02:25:53.3251108Z test_multitenancy (__main__.TCPStoreTest)
2022-11-23T02:25:53.3251427Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-11-23T02:25:53.3251712Z test_set_get (__main__.TCPStoreTest)
2022-11-23T02:25:53.3252666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3253392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3254106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3254947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3255285Z 
2022-11-23T02:25:53.3255480Z Running tests...
2022-11-23T02:25:53.3255914Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3256587Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3257369Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3258545Z ok (1.640s)
2022-11-23T02:25:53.3258808Z 
2022-11-23T02:25:53.3259283Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3259787Z Ran 1 test in 1.640s
2022-11-23T02:25:53.3260056Z 
2022-11-23T02:25:53.3260236Z OK
2022-11-23T02:25:53.3260464Z 
2022-11-23T02:25:53.3260681Z Generating XML reports...
2022-11-23T02:25:53.3261848Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022351.xml
2022-11-23T02:25:53.3262553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3263000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3263564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3264011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3264369Z 
2022-11-23T02:25:53.3264478Z Running tests...
2022-11-23T02:25:53.3264885Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3265381Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3265865Z   test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3266362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:53.3267012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:53.3267390Z ok (1.703s)
2022-11-23T02:25:53.3267536Z 
2022-11-23T02:25:53.3267801Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3268121Z Ran 1 test in 1.704s
2022-11-23T02:25:53.3268280Z 
2022-11-23T02:25:53.3268355Z OK
2022-11-23T02:25:53.3268492Z 
2022-11-23T02:25:53.3268614Z Generating XML reports...
2022-11-23T02:25:53.3269163Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022354.xml
2022-11-23T02:25:53.3269832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3270258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3270823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3271286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3271513Z 
2022-11-23T02:25:53.3271620Z Running tests...
2022-11-23T02:25:53.3272003Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3272515Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3272971Z   test_refcount (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3273279Z ok (1.664s)
2022-11-23T02:25:53.3273423Z 
2022-11-23T02:25:53.3273686Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3274003Z Ran 1 test in 1.664s
2022-11-23T02:25:53.3274243Z 
2022-11-23T02:25:53.3274324Z OK
2022-11-23T02:25:53.3274458Z 
2022-11-23T02:25:53.3274582Z Generating XML reports...
2022-11-23T02:25:53.3275124Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022358.xml
2022-11-23T02:25:53.3275786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3276210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3276776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3277237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3277466Z 
2022-11-23T02:25:53.3277559Z Running tests...
2022-11-23T02:25:53.3277956Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3278467Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3278917Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3279225Z ok (1.670s)
2022-11-23T02:25:53.3279368Z 
2022-11-23T02:25:53.3279640Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3279962Z Ran 1 test in 1.670s
2022-11-23T02:25:53.3280298Z 
2022-11-23T02:25:53.3280377Z OK
2022-11-23T02:25:53.3280514Z 
2022-11-23T02:25:53.3280638Z Generating XML reports...
2022-11-23T02:25:53.3281180Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022402.xml
2022-11-23T02:25:53.3281847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3282338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3282908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3283372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3283599Z 
2022-11-23T02:25:53.3283691Z Running tests...
2022-11-23T02:25:53.3284090Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3284603Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3285057Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3285372Z ok (1.641s)
2022-11-23T02:25:53.3285517Z 
2022-11-23T02:25:53.3285778Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3286101Z Ran 1 test in 1.642s
2022-11-23T02:25:53.3286258Z 
2022-11-23T02:25:53.3286333Z OK
2022-11-23T02:25:53.3286464Z 
2022-11-23T02:25:53.3286588Z Generating XML reports...
2022-11-23T02:25:53.3287130Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022406.xml
2022-11-23T02:25:53.3287790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3288217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3288824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3289288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3289514Z 
2022-11-23T02:25:53.3289606Z Running tests...
2022-11-23T02:25:53.3290001Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3290507Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3290961Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3291269Z ok (1.653s)
2022-11-23T02:25:53.3291416Z 
2022-11-23T02:25:53.3291734Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3292062Z Ran 1 test in 1.653s
2022-11-23T02:25:53.3292219Z 
2022-11-23T02:25:53.3292293Z OK
2022-11-23T02:25:53.3292423Z 
2022-11-23T02:25:53.3292545Z Generating XML reports...
2022-11-23T02:25:53.3293086Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022410.xml
2022-11-23T02:25:53.3293746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3294171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3294734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3295200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3295424Z 
2022-11-23T02:25:53.3295516Z Running tests...
2022-11-23T02:25:53.3295909Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3296425Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3296899Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3297226Z ok (1.658s)
2022-11-23T02:25:53.3297371Z 
2022-11-23T02:25:53.3297632Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3298323Z Ran 1 test in 1.658s
2022-11-23T02:25:53.3298486Z 
2022-11-23T02:25:53.3298562Z OK
2022-11-23T02:25:53.3298693Z 
2022-11-23T02:25:53.3298818Z Generating XML reports...
2022-11-23T02:25:53.3299387Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022414.xml
2022-11-23T02:25:53.3300160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3300581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3301146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3301601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3301825Z 
2022-11-23T02:25:53.3301916Z Running tests...
2022-11-23T02:25:53.3302311Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3302825Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3303289Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3303612Z ok (1.615s)
2022-11-23T02:25:53.3303757Z 
2022-11-23T02:25:53.3304024Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3304347Z Ran 1 test in 1.616s
2022-11-23T02:25:53.3304504Z 
2022-11-23T02:25:53.3304579Z OK
2022-11-23T02:25:53.3304709Z 
2022-11-23T02:25:53.3304832Z Generating XML reports...
2022-11-23T02:25:53.3305400Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022418.xml
2022-11-23T02:25:53.3306067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3306511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3307074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3307538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3307765Z 
2022-11-23T02:25:53.3307857Z Running tests...
2022-11-23T02:25:53.3308263Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3308775Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3309184Z   test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.003s)
2022-11-23T02:25:53.3309407Z 
2022-11-23T02:25:53.3309732Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3310059Z Ran 1 test in 0.003s
2022-11-23T02:25:53.3310218Z 
2022-11-23T02:25:53.3310310Z OK
2022-11-23T02:25:53.3310440Z 
2022-11-23T02:25:53.3310546Z Generating XML reports...
2022-11-23T02:25:53.3311099Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123022422.xml
2022-11-23T02:25:53.3311771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3312210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3312759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3313222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3313450Z 
2022-11-23T02:25:53.3313558Z Running tests...
2022-11-23T02:25:53.3313940Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3314453Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3314920Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3315249Z ok (1.621s)
2022-11-23T02:25:53.3315395Z 
2022-11-23T02:25:53.3315654Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3315972Z Ran 1 test in 1.622s
2022-11-23T02:25:53.3316129Z 
2022-11-23T02:25:53.3316221Z OK
2022-11-23T02:25:53.3316335Z 
2022-11-23T02:25:53.3316458Z Generating XML reports...
2022-11-23T02:25:53.3317021Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022424.xml
2022-11-23T02:25:53.3317763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3318187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3318751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3319206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3319430Z 
2022-11-23T02:25:53.3319539Z Running tests...
2022-11-23T02:25:53.3319920Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3320429Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3320889Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3321213Z ok (1.696s)
2022-11-23T02:25:53.3321359Z 
2022-11-23T02:25:53.3321622Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3321942Z Ran 1 test in 1.697s
2022-11-23T02:25:53.3322099Z 
2022-11-23T02:25:53.3322190Z OK
2022-11-23T02:25:53.3322305Z 
2022-11-23T02:25:53.3322432Z Generating XML reports...
2022-11-23T02:25:53.3322990Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022428.xml
2022-11-23T02:25:53.3323677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3324101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3324671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3325132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3325358Z 
2022-11-23T02:25:53.3325469Z Running tests...
2022-11-23T02:25:53.3325847Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3326358Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3326860Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3327182Z ok (1.638s)
2022-11-23T02:25:53.3327327Z 
2022-11-23T02:25:53.3327587Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3327903Z Ran 1 test in 1.638s
2022-11-23T02:25:53.3328060Z 
2022-11-23T02:25:53.3328151Z OK
2022-11-23T02:25:53.3328269Z 
2022-11-23T02:25:53.3328392Z Generating XML reports...
2022-11-23T02:25:53.3328940Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123022432.xml
2022-11-23T02:25:53.3329611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3330035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3330602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3331064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3331288Z 
2022-11-23T02:25:53.3331396Z Running tests...
2022-11-23T02:25:53.3331775Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3332287Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3332742Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3333058Z ok (1.610s)
2022-11-23T02:25:53.3333203Z 
2022-11-23T02:25:53.3333461Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3333778Z Ran 1 test in 1.610s
2022-11-23T02:25:53.3333936Z 
2022-11-23T02:25:53.3334026Z OK
2022-11-23T02:25:53.3334198Z 
2022-11-23T02:25:53.3334322Z Generating XML reports...
2022-11-23T02:25:53.3334879Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123022436.xml
2022-11-23T02:25:53.3335561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3335987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3336550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3337010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3337235Z 
2022-11-23T02:25:53.3337343Z Running tests...
2022-11-23T02:25:53.3337720Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3338679Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3339150Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3339476Z ok (1.648s)
2022-11-23T02:25:53.3339620Z 
2022-11-23T02:25:53.3339879Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3340199Z Ran 1 test in 1.648s
2022-11-23T02:25:53.3340360Z 
2022-11-23T02:25:53.3340452Z OK
2022-11-23T02:25:53.3340567Z 
2022-11-23T02:25:53.3340689Z Generating XML reports...
2022-11-23T02:25:53.3341249Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022440.xml
2022-11-23T02:25:53.3341925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3342352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3342915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3343379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3343601Z 
2022-11-23T02:25:53.3343713Z Running tests...
2022-11-23T02:25:53.3344088Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3344675Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3345143Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3345461Z ok (1.619s)
2022-11-23T02:25:53.3345604Z 
2022-11-23T02:25:53.3345864Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3346182Z Ran 1 test in 1.619s
2022-11-23T02:25:53.3346339Z 
2022-11-23T02:25:53.3346415Z OK
2022-11-23T02:25:53.3346547Z 
2022-11-23T02:25:53.3346669Z Generating XML reports...
2022-11-23T02:25:53.3347229Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022444.xml
2022-11-23T02:25:53.3347911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3348341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3348912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3349376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3349600Z 
2022-11-23T02:25:53.3349709Z Running tests...
2022-11-23T02:25:53.3350089Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3350597Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3351060Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3351386Z ok (1.679s)
2022-11-23T02:25:53.3351531Z 
2022-11-23T02:25:53.3351794Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3352205Z Ran 1 test in 1.679s
2022-11-23T02:25:53.3352363Z 
2022-11-23T02:25:53.3352438Z OK
2022-11-23T02:25:53.3352569Z 
2022-11-23T02:25:53.3352691Z Generating XML reports...
2022-11-23T02:25:53.3353257Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022447.xml
2022-11-23T02:25:53.3353932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3354358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3354924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3355386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3355613Z 
2022-11-23T02:25:53.3355721Z Running tests...
2022-11-23T02:25:53.3356101Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3356615Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3357081Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3357713Z [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-11-23T02:25:53.3358235Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-11-23T02:25:53.3358587Z ok (1.613s)
2022-11-23T02:25:53.3358733Z 
2022-11-23T02:25:53.3358997Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3359301Z Ran 1 test in 1.614s
2022-11-23T02:25:53.3359459Z 
2022-11-23T02:25:53.3359551Z OK
2022-11-23T02:25:53.3359683Z 
2022-11-23T02:25:53.3359806Z Generating XML reports...
2022-11-23T02:25:53.3360347Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022451.xml
2022-11-23T02:25:53.3361027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3361465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3362082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3362532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3362756Z 
2022-11-23T02:25:53.3362863Z Running tests...
2022-11-23T02:25:53.3363261Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3363759Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3364216Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3364554Z ok (1.652s)
2022-11-23T02:25:53.3364702Z 
2022-11-23T02:25:53.3364960Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3365268Z Ran 1 test in 1.652s
2022-11-23T02:25:53.3365427Z 
2022-11-23T02:25:53.3365519Z OK
2022-11-23T02:25:53.3365650Z 
2022-11-23T02:25:53.3365773Z Generating XML reports...
2022-11-23T02:25:53.3366313Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022455.xml
2022-11-23T02:25:53.3366987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3367429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3367991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3368432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3368662Z 
2022-11-23T02:25:53.3368770Z Running tests...
2022-11-23T02:25:53.3369167Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3369731Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3370209Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3370567Z ok (11.698s)
2022-11-23T02:25:53.3370718Z 
2022-11-23T02:25:53.3370980Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3371282Z Ran 1 test in 11.698s
2022-11-23T02:25:53.3371440Z 
2022-11-23T02:25:53.3371531Z OK
2022-11-23T02:25:53.3371663Z 
2022-11-23T02:25:53.3371788Z Generating XML reports...
2022-11-23T02:25:53.3372333Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022459.xml
2022-11-23T02:25:53.3373010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3373452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3374005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3374469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3374697Z 
2022-11-23T02:25:53.3374810Z Running tests...
2022-11-23T02:25:53.3375209Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3375707Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3376169Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3376514Z ok (1.632s)
2022-11-23T02:25:53.3376659Z 
2022-11-23T02:25:53.3376905Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3377226Z Ran 1 test in 1.633s
2022-11-23T02:25:53.3377385Z 
2022-11-23T02:25:53.3377476Z OK
2022-11-23T02:25:53.3377608Z 
2022-11-23T02:25:53.3377731Z Generating XML reports...
2022-11-23T02:25:53.3378625Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022513.xml
2022-11-23T02:25:53.3379293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3379809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3380368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3380826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3381052Z 
2022-11-23T02:25:53.3381161Z Running tests...
2022-11-23T02:25:53.3381557Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3382054Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3382524Z   test_url_with_node_params (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3382874Z ok (1.644s)
2022-11-23T02:25:53.3383018Z 
2022-11-23T02:25:53.3383260Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3383585Z Ran 1 test in 1.644s
2022-11-23T02:25:53.3383743Z 
2022-11-23T02:25:53.3383835Z OK
2022-11-23T02:25:53.3383969Z 
2022-11-23T02:25:53.3384093Z Generating XML reports...
2022-11-23T02:25:53.3384623Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022517.xml
2022-11-23T02:25:53.3385288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3385731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3386281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3386745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3387047Z 
2022-11-23T02:25:53.3387157Z Running tests...
2022-11-23T02:25:53.3387556Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3388051Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3388523Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3389168Z [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:36373 (errno: 98 - Address already in use).
2022-11-23T02:25:53.3389741Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:36373 (errno: 98 - Address already in use).
2022-11-23T02:25:53.3390194Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2022-11-23T02:25:53.3390523Z ok (1.647s)
2022-11-23T02:25:53.3390667Z 
2022-11-23T02:25:53.3390931Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3391240Z Ran 1 test in 1.647s
2022-11-23T02:25:53.3391397Z 
2022-11-23T02:25:53.3391489Z OK
2022-11-23T02:25:53.3391620Z 
2022-11-23T02:25:53.3391744Z Generating XML reports...
2022-11-23T02:25:53.3392266Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022521.xml
2022-11-23T02:25:53.3392929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3393369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3393933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3394380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3394606Z 
2022-11-23T02:25:53.3394714Z Running tests...
2022-11-23T02:25:53.3395111Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3395607Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3396066Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3396393Z ok (1.650s)
2022-11-23T02:25:53.3396538Z 
2022-11-23T02:25:53.3396853Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3397165Z Ran 1 test in 1.650s
2022-11-23T02:25:53.3397321Z 
2022-11-23T02:25:53.3397414Z OK
2022-11-23T02:25:53.3397546Z 
2022-11-23T02:25:53.3397669Z Generating XML reports...
2022-11-23T02:25:53.3398191Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022525.xml
2022-11-23T02:25:53.3398850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3399294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3399854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3400304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3400526Z 
2022-11-23T02:25:53.3400634Z Running tests...
2022-11-23T02:25:53.3401035Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3401528Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3402011Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3402505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:53.3403155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:53.3403528Z ok (1.736s)
2022-11-23T02:25:53.3403673Z 
2022-11-23T02:25:53.3403934Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3404365Z Ran 1 test in 1.736s
2022-11-23T02:25:53.3404523Z 
2022-11-23T02:25:53.3404598Z OK
2022-11-23T02:25:53.3404728Z 
2022-11-23T02:25:53.3404852Z Generating XML reports...
2022-11-23T02:25:53.3405397Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022529.xml
2022-11-23T02:25:53.3406056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3406479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3407041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3407501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3407725Z 
2022-11-23T02:25:53.3407833Z Running tests...
2022-11-23T02:25:53.3408212Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3408724Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3409209Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3409545Z ok (1.685s)
2022-11-23T02:25:53.3409690Z 
2022-11-23T02:25:53.3409953Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3410277Z Ran 1 test in 1.686s
2022-11-23T02:25:53.3410435Z 
2022-11-23T02:25:53.3410510Z OK
2022-11-23T02:25:53.3410640Z 
2022-11-23T02:25:53.3410762Z Generating XML reports...
2022-11-23T02:25:53.3411299Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022533.xml
2022-11-23T02:25:53.3411953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3412377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3412942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3413412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3413638Z 
2022-11-23T02:25:53.3413745Z Running tests...
2022-11-23T02:25:53.3414178Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3414700Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3415184Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3415524Z ok (1.666s)
2022-11-23T02:25:53.3415668Z 
2022-11-23T02:25:53.3415927Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3416247Z Ran 1 test in 1.666s
2022-11-23T02:25:53.3416403Z 
2022-11-23T02:25:53.3416478Z OK
2022-11-23T02:25:53.3416609Z 
2022-11-23T02:25:53.3416730Z Generating XML reports...
2022-11-23T02:25:53.3417273Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022537.xml
2022-11-23T02:25:53.3418348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3418788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3419354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3419812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3420035Z 
2022-11-23T02:25:53.3420126Z Running tests...
2022-11-23T02:25:53.3420523Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3421033Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3421489Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3421896Z ok (1.647s)
2022-11-23T02:25:53.3422041Z 
2022-11-23T02:25:53.3422304Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3422624Z Ran 1 test in 1.647s
2022-11-23T02:25:53.3422782Z 
2022-11-23T02:25:53.3422857Z OK
2022-11-23T02:25:53.3422989Z 
2022-11-23T02:25:53.3423115Z Generating XML reports...
2022-11-23T02:25:53.3423653Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022541.xml
2022-11-23T02:25:53.3424314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3424738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3425303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3425766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3425993Z 
2022-11-23T02:25:53.3426088Z Running tests...
2022-11-23T02:25:53.3426484Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3426994Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3427454Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3427774Z ok (3.668s)
2022-11-23T02:25:53.3427918Z 
2022-11-23T02:25:53.3428179Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3428494Z Ran 1 test in 3.668s
2022-11-23T02:25:53.3428653Z 
2022-11-23T02:25:53.3428727Z OK
2022-11-23T02:25:53.3428859Z 
2022-11-23T02:25:53.3428985Z Generating XML reports...
2022-11-23T02:25:53.3429521Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022545.xml
2022-11-23T02:25:53.3430185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:53.3430619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:53.3431182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:53.3431714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:53.3431949Z 
2022-11-23T02:25:53.3432040Z Running tests...
2022-11-23T02:25:53.3432437Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3432949Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:25:53.3433394Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:53.3433698Z ok (1.661s)
2022-11-23T02:25:53.3433843Z 
2022-11-23T02:25:53.3434104Z ----------------------------------------------------------------------
2022-11-23T02:25:53.3434426Z Ran 1 test in 1.661s
2022-11-23T02:25:53.3434587Z 
2022-11-23T02:25:53.3434663Z OK
2022-11-23T02:25:53.3434799Z 
2022-11-23T02:25:53.3434922Z Generating XML reports...
2022-11-23T02:25:53.3435462Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022550.xml
2022-11-23T02:25:53.3435775Z 
2022-11-23T02:25:53.3436130Z ##[endgroup]
2022-11-23T02:25:53.3436673Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_4zf70t7m)
2022-11-23T02:25:53.3436985Z 
2022-11-23T02:25:53.3437255Z Running distributed/_tensor/test_tensor_ops ... [2022-11-23 02:25:53.322886]
2022-11-23T02:25:53.3437924Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:25:53.323176]
2022-11-23T02:27:09.4022948Z 
2022-11-23T02:27:09.4023597Z Expand the folded group to see the log file of distributed/_tensor/test_tensor_ops
2022-11-23T02:27:09.4024659Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_wre2ccot)
2022-11-23T02:27:09.4027855Z 
2022-11-23T02:27:09.4028271Z Running tests...
2022-11-23T02:27:09.4028835Z ----------------------------------------------------------------------
2022-11-23T02:27:09.4029428Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tensor_ops
2022-11-23T02:27:09.4029932Z   test_aten_contiguous (__main__.DistTensorOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:09.4030384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57274
2022-11-23T02:27:09.4030834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57275
2022-11-23T02:27:09.4031452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4032361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4032935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4033412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4034006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4034453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4035003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4035467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4035904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4036373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4036885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4037370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4038028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4038896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4039296Z ok (5.629s)
2022-11-23T02:27:09.4039703Z   test_clone (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57355
2022-11-23T02:27:09.4040213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57356
2022-11-23T02:27:09.4040810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4041261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4041834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4042298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4042856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4043302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4043864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4044326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4044742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4045226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4045702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4046238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4046899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4047585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4047977Z ok (4.111s)
2022-11-23T02:27:09.4048371Z   test_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57436
2022-11-23T02:27:09.4048876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57437
2022-11-23T02:27:09.4049479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4049906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4050484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4050948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4051524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4051946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4052511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4052970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4053401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4053861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4054340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4054826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4055509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4056205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4056594Z ok (4.011s)
2022-11-23T02:27:09.4057000Z   test_detach (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57519
2022-11-23T02:27:09.4057478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57520
2022-11-23T02:27:09.4058562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4059033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4059598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4060064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4060639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4061081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4061631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4062100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4062534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4063017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4063590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4064071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4064728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4065394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4065786Z ok (4.010s)
2022-11-23T02:27:09.4066285Z   test_empty_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57600
2022-11-23T02:27:09.4066794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57601
2022-11-23T02:27:09.4067403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4067852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4068404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4068869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4069445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4069883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4070434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4070893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4071330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4071790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4072275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4072759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4073477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4074162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4074554Z ok (4.011s)
2022-11-23T02:27:09.4074968Z   test_fill_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57679
2022-11-23T02:27:09.4075455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57680
2022-11-23T02:27:09.4076056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4076508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4077077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4077526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4078097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4078538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4079102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4079545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4079978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4080518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4080980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4081463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4082112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4082795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4083166Z ok (4.011s)
2022-11-23T02:27:09.4083598Z   test_fill_inplace_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57758
2022-11-23T02:27:09.4084124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57759
2022-11-23T02:27:09.4084714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4085158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4085730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4086192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4086740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4087182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4087744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4088202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4088660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4089147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4089625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4090139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4090797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4091472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4091864Z ok (3.911s)
2022-11-23T02:27:09.4092258Z   test_full_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57837
2022-11-23T02:27:09.4092766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57838
2022-11-23T02:27:09.4093374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4093817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4094370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4094832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4095406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4095827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4096396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4096854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4097344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4097807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4098695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4099180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4099815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4100506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4100898Z ok (3.910s)
2022-11-23T02:27:09.4101311Z   test_index (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57916
2022-11-23T02:27:09.4101793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57917
2022-11-23T02:27:09.4102401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4102849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4103419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4103865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4104439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4104885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4105429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4105894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4106331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4106816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4107366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4107858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4108510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4109188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4109562Z ok (12.229s)
2022-11-23T02:27:09.4109977Z   test_inplace_op (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58027
2022-11-23T02:27:09.4110490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58028
2022-11-23T02:27:09.4111074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4111521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4112089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4112555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4113107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4113546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4114107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4114549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4115058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4115524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4116004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4116472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4117121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4117803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4118193Z ok (4.011s)
2022-11-23T02:27:09.4118585Z   test_ones_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58108
2022-11-23T02:27:09.4119097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58109
2022-11-23T02:27:09.4119704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4120129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4120696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4121160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4121733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4122154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4122721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4123186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4123598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4124133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4124620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4125100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4125728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4126400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4126789Z ok (3.910s)
2022-11-23T02:27:09.4127217Z   test_ones_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58187
2022-11-23T02:27:09.4127720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58188
2022-11-23T02:27:09.4128322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4128771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4129322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4129782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4130354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4130794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4131341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4131877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4132312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4132781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4133264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4133742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4134391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4135055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4135445Z ok (4.011s)
2022-11-23T02:27:09.4135868Z   test_op_out_variant (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58266
2022-11-23T02:27:09.4136381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58267
2022-11-23T02:27:09.4136970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4137418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4138392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4138861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4139443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4139882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4140448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4140898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4141330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4141897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4142392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4142849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4143495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4144177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4144555Z ok (4.011s)
2022-11-23T02:27:09.4144969Z   test_zero_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58347
2022-11-23T02:27:09.4145476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58348
2022-11-23T02:27:09.4146081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4146509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4147077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4147543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4148098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4148539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4149180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4149641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4150058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4150537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4151014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4151486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4152112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4152789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4153182Z ok (3.910s)
2022-11-23T02:27:09.4153574Z   test_zeros_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58426
2022-11-23T02:27:09.4154081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58427
2022-11-23T02:27:09.4154684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4155127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4155674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4156137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4156710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4157149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4157699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4158161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4158643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4159118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4159596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4160073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4160722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4161384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4161780Z ok (4.010s)
2022-11-23T02:27:09.4162209Z   test_zeros_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58505
2022-11-23T02:27:09.4162714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58506
2022-11-23T02:27:09.4163313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4163754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4164328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4164777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4165349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:09.4165853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:09.4166420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:09.4166870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:09.4167301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:09.4167782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:09.4168244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:09.4168723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:09.4169368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4170054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:09.4170424Z ok (4.010s)
2022-11-23T02:27:09.4170573Z 
2022-11-23T02:27:09.4170842Z ----------------------------------------------------------------------
2022-11-23T02:27:09.4171174Z Ran 16 tests in 73.708s
2022-11-23T02:27:09.4171338Z 
2022-11-23T02:27:09.4171412Z OK
2022-11-23T02:27:09.4171546Z 
2022-11-23T02:27:09.4171670Z Generating XML reports...
2022-11-23T02:27:09.4172254Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20221123022555.xml
2022-11-23T02:27:09.4172598Z 
2022-11-23T02:27:09.4173030Z ##[endgroup]
2022-11-23T02:27:09.4173606Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_wre2ccot)
2022-11-23T02:27:09.4173953Z 
2022-11-23T02:27:09.4174209Z Running distributed/fsdp/test_wrap ... [2022-11-23 02:27:09.402449]
2022-11-23T02:27:09.4174863Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:27:09.402713]
2022-11-23T02:28:51.2396385Z 
2022-11-23T02:28:51.2397454Z Expand the folded group to see the log file of distributed/fsdp/test_wrap
2022-11-23T02:28:51.2398385Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_gf5rjk0i)
2022-11-23T02:28:51.2398733Z 
2022-11-23T02:28:51.2398854Z Running tests...
2022-11-23T02:28:51.2399360Z ----------------------------------------------------------------------
2022-11-23T02:28:51.2399907Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2022-11-23T02:28:51.2400319Z   test_always_wrap (__main__.TestAutoWrap)
2022-11-23T02:28:51.2400703Z Test to ensure that if `always_wrap_policy` is ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:51.2401059Z ok (1.643s)
2022-11-23T02:28:51.2404847Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2405815Z   warnings.warn(
2022-11-23T02:28:51.2406072Z ok (0.005s)
2022-11-23T02:28:51.2406436Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.005s)
2022-11-23T02:28:51.2406849Z   test_auto_wrap_api (__main__.TestAutoWrap)
2022-11-23T02:28:51.2407267Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... ok (0.003s)
2022-11-23T02:28:51.2407864Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2022-11-23T02:28:51.2408323Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... ok (0.002s)
2022-11-23T02:28:51.2408818Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2022-11-23T02:28:51.2409288Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... ok (0.002s)
2022-11-23T02:28:51.2409702Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2022-11-23T02:28:51.2410247Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... ok (0.004s)
2022-11-23T02:28:51.2410682Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2022-11-23T02:28:51.2411138Z Test to ensure force-leaf modules are not wrapped. ... ok (0.002s)
2022-11-23T02:28:51.2411778Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2412644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:28:51.2413040Z ok (0.467s)
2022-11-23T02:28:51.2413631Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2414464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:28:51.2414846Z ok (0.043s)
2022-11-23T02:28:51.2415294Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:28:51.2415938Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:28:51.2416767Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2417608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:28:51.2419067Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:28:51.2420327Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:28:51.2420917Z ok (0.060s)
2022-11-23T02:28:51.2421499Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2422322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:28:51.2422716Z ok (0.042s)
2022-11-23T02:28:51.2423289Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2424223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:28:51.2424596Z ok (0.043s)
2022-11-23T02:28:51.2425167Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2426269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:28:51.2428159Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:28:51.2430683Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:28:51.2433239Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:28:51.2434547Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:28:51.2435149Z ok (0.074s)
2022-11-23T02:28:51.2435509Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.003s)
2022-11-23T02:28:51.2436104Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.003s)
2022-11-23T02:28:51.2436535Z   test_module_wrap_policy (__main__.TestAutoWrap)
2022-11-23T02:28:51.2436858Z Tests the ``ModuleWrapPolicy``. ... ok (0.026s)
2022-11-23T02:28:51.2437211Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap)
2022-11-23T02:28:51.2437579Z Tests the ``transformer_auto_wrap_policy``. ... ok (0.020s)
2022-11-23T02:28:51.2437948Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:28:51.2438347Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:28:51.2438762Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:28:51.2439200Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:28:51.2439586Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2022-11-23T02:28:51.2440104Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58651
2022-11-23T02:28:51.2440643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58652
2022-11-23T02:28:51.2441240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2441688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2442258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2442726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2443344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2443789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2444362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2444826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2445252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2445743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2446394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2447059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2447578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2448045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2449309Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2450077Z   warnings.warn(
2022-11-23T02:28:51.2451207Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2451983Z   warnings.warn(
2022-11-23T02:28:51.2452283Z dist init r=1, world=2
2022-11-23T02:28:51.2452541Z dist init r=0, world=2
2022-11-23T02:28:51.2452761Z ok (3.913s)
2022-11-23T02:28:51.2453141Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-11-23T02:28:51.2453713Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58730
2022-11-23T02:28:51.2454230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58731
2022-11-23T02:28:51.2454839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2455291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2455861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2456311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2456886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2457331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2458286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2458748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2459197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2459690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2460420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2461109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2461631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2462098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2463338Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2464120Z   warnings.warn(
2022-11-23T02:28:51.2465262Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2466033Z   warnings.warn(
2022-11-23T02:28:51.2466287Z dist init r=0, world=2
2022-11-23T02:28:51.2466518Z dist init r=1, world=2
2022-11-23T02:28:51.2466756Z ok (4.012s)
2022-11-23T02:28:51.2467136Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-11-23T02:28:51.2467683Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58809
2022-11-23T02:28:51.2468221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58810
2022-11-23T02:28:51.2468890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2469350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2469901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2470368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2470941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2471385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2471935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2472403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2472853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2473327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2473976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2474659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2475180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2475629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2475982Z dist init r=0, world=2
2022-11-23T02:28:51.2476287Z dist init r=1, world=2
2022-11-23T02:28:51.2476510Z ok (3.910s)
2022-11-23T02:28:51.2476885Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-11-23T02:28:51.2477456Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58888
2022-11-23T02:28:51.2477986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58889
2022-11-23T02:28:51.2478577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2479023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2479595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2480042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2480620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2481061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2481636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2482082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2482530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2483023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2483671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2484336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2484851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2485316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2486616Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2487369Z   warnings.warn(
2022-11-23T02:28:51.2488565Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2489332Z   warnings.warn(
2022-11-23T02:28:51.2489582Z dist init r=1, world=2
2022-11-23T02:28:51.2489817Z dist init r=0, world=2
2022-11-23T02:28:51.2490056Z ok (3.911s)
2022-11-23T02:28:51.2490430Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-11-23T02:28:51.2490998Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58967
2022-11-23T02:28:51.2491511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58968
2022-11-23T02:28:51.2492114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2492617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2493169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2493638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2494214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2494656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2495207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2495667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2496114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2496586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2497245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2498219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2498749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2499198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2499551Z dist init r=1, world=2
2022-11-23T02:28:51.2499803Z dist init r=0, world=2
2022-11-23T02:28:51.2500026Z ok (3.911s)
2022-11-23T02:28:51.2500652Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59046
2022-11-23T02:28:51.2501362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59047
2022-11-23T02:28:51.2501972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2502486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2503069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2503538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2504116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2504537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2505107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2505572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2506021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2506497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2507147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2507834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2508330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2508797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2510047Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2510896Z   warnings.warn(
2022-11-23T02:28:51.2512033Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2512798Z   warnings.warn(
2022-11-23T02:28:51.2513030Z dist init r=1, world=2
2022-11-23T02:28:51.2513285Z dist init r=0, world=2
2022-11-23T02:28:51.2513525Z ok (4.514s)
2022-11-23T02:28:51.2514138Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59129
2022-11-23T02:28:51.2514844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59130
2022-11-23T02:28:51.2515452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2515899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2516455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2516920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2517499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2517924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2518538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2519006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2519461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2519933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2520584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2521269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2521788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2522234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2522578Z dist init r=1, world=2
2022-11-23T02:28:51.2522835Z dist init r=0, world=2
2022-11-23T02:28:51.2523058Z ok (4.413s)
2022-11-23T02:28:51.2523684Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59212
2022-11-23T02:28:51.2524390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59213
2022-11-23T02:28:51.2524994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2525422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2526049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2526512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2527074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2527514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2528081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2528543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2528976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2529466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2530124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2530807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2531307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2531776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2533032Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2533813Z   warnings.warn(
2022-11-23T02:28:51.2534993Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2535761Z   warnings.warn(
2022-11-23T02:28:51.2536013Z dist init r=0, world=2
2022-11-23T02:28:51.2536265Z dist init r=1, world=2
2022-11-23T02:28:51.2536485Z ok (4.413s)
2022-11-23T02:28:51.2537107Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59295
2022-11-23T02:28:51.2537820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59296
2022-11-23T02:28:51.2538925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2539357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2539928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2540395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2540964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2541385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2541952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2542491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2542921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2543416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2544069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2544757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2545258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2545726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2546084Z dist init r=1, world=2
2022-11-23T02:28:51.2546316Z dist init r=0, world=2
2022-11-23T02:28:51.2546557Z ok (4.413s)
2022-11-23T02:28:51.2547190Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59378
2022-11-23T02:28:51.2547896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59379
2022-11-23T02:28:51.2548488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2548934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2549504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2549967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2550525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2550975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2551540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2552099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2552559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2553047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2553700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2554362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2554880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2555352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2556613Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2557378Z   warnings.warn(
2022-11-23T02:28:51.2558512Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2559330Z   warnings.warn(
2022-11-23T02:28:51.2559585Z dist init r=0, world=2
2022-11-23T02:28:51.2559838Z dist init r=1, world=2
2022-11-23T02:28:51.2560069Z ok (4.513s)
2022-11-23T02:28:51.2560702Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59461
2022-11-23T02:28:51.2561406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59462
2022-11-23T02:28:51.2562021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2562451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2563027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2563492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2564050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2564490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2565058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2565518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2565943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2566426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2567072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2567778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2568327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2568802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2569155Z dist init r=0, world=2
2022-11-23T02:28:51.2569387Z dist init r=1, world=2
2022-11-23T02:28:51.2569624Z ok (4.413s)
2022-11-23T02:28:51.2570251Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59544
2022-11-23T02:28:51.2570959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59545
2022-11-23T02:28:51.2571556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2572004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2572577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2573046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2573596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2574038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2574614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2575072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2575583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2576086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2576745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2577411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2578235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2578709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2579976Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2580756Z   warnings.warn(
2022-11-23T02:28:51.2581891Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2582650Z   warnings.warn(
2022-11-23T02:28:51.2582898Z dist init r=0, world=2
2022-11-23T02:28:51.2583144Z dist init r=1, world=2
2022-11-23T02:28:51.2583365Z ok (4.513s)
2022-11-23T02:28:51.2583990Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59627
2022-11-23T02:28:51.2584771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59628
2022-11-23T02:28:51.2585391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2585819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2586388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2586851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2587407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2587850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2588451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2588918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2589347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2589836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2590486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2591168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2591665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2592206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2592581Z dist init r=1, world=2
2022-11-23T02:28:51.2592818Z dist init r=0, world=2
2022-11-23T02:28:51.2593054Z ok (4.513s)
2022-11-23T02:28:51.2593689Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59710
2022-11-23T02:28:51.2594407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59711
2022-11-23T02:28:51.2595006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2595669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2596260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2596734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2597293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2597738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2598307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2598768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2599196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2599683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2600332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2601014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2601511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2602035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2602393Z dist init r=1, world=2
2022-11-23T02:28:51.2602625Z dist init r=0, world=2
2022-11-23T02:28:51.2602863Z ok (4.012s)
2022-11-23T02:28:51.2603497Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59789
2022-11-23T02:28:51.2604206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59790
2022-11-23T02:28:51.2604823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2605287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2605876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2606344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2606903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2607339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2607909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2608447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2608899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2609443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2610083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2610761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2611277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2611745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2612078Z dist init r=0, world=2
2022-11-23T02:28:51.2612329Z dist init r=1, world=2
2022-11-23T02:28:51.2612590Z ok (4.514s)
2022-11-23T02:28:51.2613230Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59872
2022-11-23T02:28:51.2613922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59873
2022-11-23T02:28:51.2614527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2614998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2615571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2616016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2616589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2617034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2617622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2618379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2618922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2619419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2620082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2620747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2621259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2621727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2622083Z dist init r=1, world=2
2022-11-23T02:28:51.2622315Z dist init r=0, world=2
2022-11-23T02:28:51.2622553Z ok (4.014s)
2022-11-23T02:28:51.2623181Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59951
2022-11-23T02:28:51.2623873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59952
2022-11-23T02:28:51.2624478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2624920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2625490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2625936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2626592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2627037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2627602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2628046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2628494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2628993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2629626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2630306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2630830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2631295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2631635Z dist init r=0, world=2
2022-11-23T02:28:51.2631885Z dist init r=1, world=2
2022-11-23T02:28:51.2632125Z ok (4.413s)
2022-11-23T02:28:51.2632727Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60034
2022-11-23T02:28:51.2633433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60035
2022-11-23T02:28:51.2634038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2634489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2635041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2635545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2636128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2636563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2637112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2637575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2638028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2638497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2639163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2639853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2640373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2640821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2641168Z dist init r=0, world=2
2022-11-23T02:28:51.2641418Z dist init r=1, world=2
2022-11-23T02:28:51.2641641Z ok (3.912s)
2022-11-23T02:28:51.2642264Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60113
2022-11-23T02:28:51.2643033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60114
2022-11-23T02:28:51.2643640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2644070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2644640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2645105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2645678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2646099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2646663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2647130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2647558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2648052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2648704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2649387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2649885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2650348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2650701Z dist init r=1, world=2
2022-11-23T02:28:51.2650935Z dist init r=0, world=2
2022-11-23T02:28:51.2651178Z ok (4.513s)
2022-11-23T02:28:51.2651844Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60196
2022-11-23T02:28:51.2652553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60197
2022-11-23T02:28:51.2653140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2653587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2654156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2654620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2655172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2655620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2656189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2656633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2657080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2657570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2658597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2659263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2659787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2660340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2660692Z dist init r=1, world=2
2022-11-23T02:28:51.2660925Z dist init r=0, world=2
2022-11-23T02:28:51.2661168Z ok (3.912s)
2022-11-23T02:28:51.2661792Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60275
2022-11-23T02:28:51.2662483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60276
2022-11-23T02:28:51.2663091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2663538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2664114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2664560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2665139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2665578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2666148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2666589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2667035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2667529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2668159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2668848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2669435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2669914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2670248Z dist init r=0, world=2
2022-11-23T02:28:51.2670500Z dist init r=1, world=2
2022-11-23T02:28:51.2670746Z ok (4.513s)
2022-11-23T02:28:51.2671183Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60358
2022-11-23T02:28:51.2671728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60359
2022-11-23T02:28:51.2672332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2672783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2673335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2673802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2674377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2674798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2675367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2675826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2676274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2676967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2677497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2678143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2678661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2679107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2680362Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2681134Z   warnings.warn(
2022-11-23T02:28:51.2682288Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2683056Z   warnings.warn(
2022-11-23T02:28:51.2683287Z dist init r=1, world=2
2022-11-23T02:28:51.2683537Z dist init r=0, world=2
2022-11-23T02:28:51.2683778Z ok (3.911s)
2022-11-23T02:28:51.2684213Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60437
2022-11-23T02:28:51.2684762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60438
2022-11-23T02:28:51.2685373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2685867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2686429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2686891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2687461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:51.2687881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:51.2688490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:51.2688953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:51.2689399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:51.2689870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:51.2690512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2691195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:51.2691712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:51.2692161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:51.2693416Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2694240Z   warnings.warn(
2022-11-23T02:28:51.2695390Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:51.2696146Z   warnings.warn(
2022-11-23T02:28:51.2696377Z dist init r=1, world=2
2022-11-23T02:28:51.2696631Z dist init r=0, world=2
2022-11-23T02:28:51.2696867Z ok (3.910s)
2022-11-23T02:28:51.2697017Z 
2022-11-23T02:28:51.2697270Z ----------------------------------------------------------------------
2022-11-23T02:28:51.2697599Z Ran 47 tests in 99.451s
2022-11-23T02:28:51.2697761Z 
2022-11-23T02:28:51.2697855Z OK
2022-11-23T02:28:51.2698313Z 
2022-11-23T02:28:51.2698424Z Generating XML reports...
2022-11-23T02:28:51.2698989Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123022711.xml
2022-11-23T02:28:51.2699696Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123022711.xml
2022-11-23T02:28:51.2700022Z 
2022-11-23T02:28:51.2700416Z ##[endgroup]
2022-11-23T02:28:51.2700964Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_gf5rjk0i)
2022-11-23T02:28:51.2701300Z 
2022-11-23T02:28:51.2701607Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2022-11-23 02:28:51.240023]
2022-11-23T02:28:51.2702341Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:28:51.240295]
2022-11-23T02:31:22.8713324Z 
2022-11-23T02:31:22.8714597Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T02:31:22.8718086Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_igf6e9bi)
2022-11-23T02:31:22.8718532Z 
2022-11-23T02:31:22.8718651Z Running tests...
2022-11-23T02:31:22.8719171Z ----------------------------------------------------------------------
2022-11-23T02:31:22.8719771Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor
2022-11-23T02:31:22.8720457Z   test_empty (__main__.TestCreateTensorFromParams) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:31:22.8721111Z ok (1.643s)
2022-11-23T02:31:22.8721807Z   test_local_tensor (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60551
2022-11-23T02:31:22.8722778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60552
2022-11-23T02:31:22.8723505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60553
2022-11-23T02:31:22.8723953Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60554
2022-11-23T02:31:22.8724603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8725250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8726587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8727324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8727911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8730555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8731235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8731737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8732325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8732778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8733337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8733810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8734400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8734832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8735688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8736170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8736610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8737073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8737538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8738361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8738972Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.8739686Z   test_local_tensor_error (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60687
2022-11-23T02:31:22.8740276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60688
2022-11-23T02:31:22.8740898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60689
2022-11-23T02:31:22.8741364Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60690
2022-11-23T02:31:22.8741994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8742429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8743007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8743481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8744067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8744496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8745069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8745538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8746096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8746540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8747114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8747579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8748138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8748679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8749255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8749717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8750138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8750608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8751080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8751521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8751907Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8752384Z   test_collect_local_shard (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60823
2022-11-23T02:31:22.8752911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60824
2022-11-23T02:31:22.8753342Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60825
2022-11-23T02:31:22.8753776Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60826
2022-11-23T02:31:22.8754387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8754837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8755400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8755871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8756453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8756880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8757496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8757943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8758520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8758966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8759550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8760007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8760579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8761012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8761589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8762055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8762473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8762947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8763410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8763872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8764239Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8764700Z   test_reshard_output (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60959
2022-11-23T02:31:22.8765271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60960
2022-11-23T02:31:22.8765695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60961
2022-11-23T02:31:22.8766139Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60962
2022-11-23T02:31:22.8766740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8767183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8767733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8768194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8768769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8769212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8769761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8770223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8770790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8771208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8771777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8772233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8772796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8773213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8773775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8774228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8774689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8775168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8775623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8776086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8776455Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8776937Z   test_create_shard_with_no_placement (__main__.TestShardMetadata) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61095
2022-11-23T02:31:22.8777475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61096
2022-11-23T02:31:22.8778305Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61097
2022-11-23T02:31:22.8778753Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61098
2022-11-23T02:31:22.8779362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8779807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8780343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8780784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8781354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8781820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8782473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8782931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8783506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8783928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8784493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8784952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8785523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8785942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8786515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8786975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8787452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8787902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8788356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8788817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8789187Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:31:22.8789655Z   test_shard_metadata_init (__main__.TestShardMetadata) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61231
2022-11-23T02:31:22.8790170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61232
2022-11-23T02:31:22.8790614Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61233
2022-11-23T02:31:22.8791034Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61234
2022-11-23T02:31:22.8791705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8792161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8792712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8793176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8793749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8794186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8794736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8795198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8795773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8796192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8796760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8797214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8797788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8798206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8798774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8799288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8799721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8800176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8800633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8801098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8801470Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8801939Z   test_shard_parameter (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61367
2022-11-23T02:31:22.8802463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61368
2022-11-23T02:31:22.8802913Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61369
2022-11-23T02:31:22.8803337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61370
2022-11-23T02:31:22.8803942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8804386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8804936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8805398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8805970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8806406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8806953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8807416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8807983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8808471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8809027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8809482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8810049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8810469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8811037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8811500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8811930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8812381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8812835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8813295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8813662Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8814139Z   test_shard_parameter_errors (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61503
2022-11-23T02:31:22.8815207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61504
2022-11-23T02:31:22.8815654Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61505
2022-11-23T02:31:22.8816150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61506
2022-11-23T02:31:22.8816752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8817203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8817767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8818542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8819121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8819561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8820106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8820569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8821138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8821577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8822122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8822586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8823152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8823570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8824131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8824584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8825020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8825467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8825994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8826466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8826854Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8827287Z   test_shard_tensor (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61639
2022-11-23T02:31:22.8827786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61640
2022-11-23T02:31:22.8828225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61641
2022-11-23T02:31:22.8828643Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61642
2022-11-23T02:31:22.8829513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8830021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8830602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8831050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8831618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8832058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8832594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8833036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8833709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8834171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8834736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8835192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8835767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8836204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8836752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8837206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8837643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8838090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8838543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8838996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8839379Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8839821Z   test_shard_tensor_errors (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61775
2022-11-23T02:31:22.8840325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61776
2022-11-23T02:31:22.8840759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61777
2022-11-23T02:31:22.8841177Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61778
2022-11-23T02:31:22.8841780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8842225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8842851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8843284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8843853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8844315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8844875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8845333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8845904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8846350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8846898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8847359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8847929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8848364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8848908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8849363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8849794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8850302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8850758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8851285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8851754Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8852244Z   test_cleanup (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61911
2022-11-23T02:31:22.8852916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61912
2022-11-23T02:31:22.8853447Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61913
2022-11-23T02:31:22.8853902Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61914
2022-11-23T02:31:22.8854587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8855109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8855808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8856297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8856947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8875759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8876462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8876929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8877507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8877970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8878543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8879125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8879724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8880170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8880719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8881175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8881606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8882079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8882536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8882994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8883390Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8883863Z   test_complete_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62047
2022-11-23T02:31:22.8884402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62048
2022-11-23T02:31:22.8884848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62049
2022-11-23T02:31:22.8885293Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62050
2022-11-23T02:31:22.8885889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8886444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8887017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8887544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8888103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8888538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8889098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8889549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8890121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8890564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8891125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8891572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8892148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8892586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8893145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8893594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8894030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8894498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8894947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8895407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8895856Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.8896238Z   test_create_sharded_tensor_like (__main__.TestShardedTensorChunked)
2022-11-23T02:31:22.8896753Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62183
2022-11-23T02:31:22.8897289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62184
2022-11-23T02:31:22.8897728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62185
2022-11-23T02:31:22.8898593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62186
2022-11-23T02:31:22.8899207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8899664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8900238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8900694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8901267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8901707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8902254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8902716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8903286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8903826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8904373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8904837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8905407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8905848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8906395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8906857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8907289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8907743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8908201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8908670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8909064Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.8909425Z   test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked)
2022-11-23T02:31:22.8909910Z Test sharded_tensor.full(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62319
2022-11-23T02:31:22.8910397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62320
2022-11-23T02:31:22.8910827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62321
2022-11-23T02:31:22.8911267Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62322
2022-11-23T02:31:22.8911874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8912327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8912949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8913425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8913997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8914419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8914982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8915443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8916014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8916441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8917002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8917466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8918041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8918465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8919027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8919490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8919904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8920375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8920965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8921431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8921806Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:31:22.8922187Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked)
2022-11-23T02:31:22.8922674Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62455
2022-11-23T02:31:22.8923143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62456
2022-11-23T02:31:22.8923592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62457
2022-11-23T02:31:22.8924037Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62458
2022-11-23T02:31:22.8924644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8925081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8925655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8926122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8926676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8927115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8927675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8928138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8928687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8929135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8929697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8930213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8930778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8931220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8931779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8932222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8932657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8933132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8933594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8934038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8934423Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.8934802Z   test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked)
2022-11-23T02:31:22.8935280Z Test sharded_tensor.rand(...)/randn(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62591
2022-11-23T02:31:22.8935775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62592
2022-11-23T02:31:22.8936221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62593
2022-11-23T02:31:22.8936664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62594
2022-11-23T02:31:22.8937253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8937760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8938504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8938935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8939502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8939971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8940554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8941003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8941573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8942020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8942584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8943032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8943605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8944046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8944591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8945048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8945480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8945955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8946402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8946940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8947337Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:31:22.8947701Z   test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked)
2022-11-23T02:31:22.8948186Z Test sharded_tensor.zeros(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62727
2022-11-23T02:31:22.8948673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62728
2022-11-23T02:31:22.8949117Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62729
2022-11-23T02:31:22.8949545Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62730
2022-11-23T02:31:22.8950152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8950600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8951160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8951628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8952201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8952645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8953194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8953655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8954222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8954738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8955290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8955749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8956318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8956736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8957297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8957753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8958185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8958639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8959095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8959567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8959940Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8960294Z   test_gather_even (__main__.TestShardedTensorChunked)
2022-11-23T02:31:22.8960803Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62863
2022-11-23T02:31:22.8961339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62864
2022-11-23T02:31:22.8961768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62865
2022-11-23T02:31:22.8962202Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62866
2022-11-23T02:31:22.8962815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8963245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8963870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8964349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8964925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8965351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8965914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8966378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8966949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8967374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8967936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8968399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8968956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8969393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8969954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8970412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8970829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8971361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8971825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8972271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8972661Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8973015Z   test_gather_uneven (__main__.TestShardedTensorChunked)
2022-11-23T02:31:22.8973528Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62999
2022-11-23T02:31:22.8974045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63000
2022-11-23T02:31:22.8974490Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63001
2022-11-23T02:31:22.8974936Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63002
2022-11-23T02:31:22.8975531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8975981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8976549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8977019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8977576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8978253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8978834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8979299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8979859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8980304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8980956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8981413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8981991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8982430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8982988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8983431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8983865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8984341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8984786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8985250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8985635Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8986133Z   test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63135
2022-11-23T02:31:22.8986669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63136
2022-11-23T02:31:22.8987114Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63137
2022-11-23T02:31:22.8987598Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63138
2022-11-23T02:31:22.8988284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8988712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8989284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8989749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8990306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8990750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8991316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8991778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8992326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8992772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8993336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8993781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8994353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.8994794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.8995357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.8995796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.8996234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.8996709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.8997170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.8997672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.8998065Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.8998553Z   test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63271
2022-11-23T02:31:22.8999079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63272
2022-11-23T02:31:22.8999523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63273
2022-11-23T02:31:22.8999971Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63274
2022-11-23T02:31:22.9000581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9001018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9001588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9002060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9002616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9003062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9003629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9004095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9004649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9005164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9005734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9006198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9006754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9007195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9007761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9008202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9008634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9009104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9009573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9010023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9010410Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9010893Z   test_invalid_sharding (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63407
2022-11-23T02:31:22.9011411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63408
2022-11-23T02:31:22.9011855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63409
2022-11-23T02:31:22.9012297Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63410
2022-11-23T02:31:22.9012904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9013340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9013892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9014386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9014945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9015409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9015989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9016455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9017008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9017452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9018182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9018653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9019215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9019657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9020218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9020659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9021090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9021564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9022111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9022560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9022946Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:31:22.9023440Z   test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63543
2022-11-23T02:31:22.9023965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63544
2022-11-23T02:31:22.9024413Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63545
2022-11-23T02:31:22.9024856Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63546
2022-11-23T02:31:22.9025459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9025889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9026467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9026933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9027507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9027927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9028493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9028954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9029505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9029998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9030572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9031032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9031656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9032106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9032671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9033109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9033540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9034011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9034474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9034924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9035310Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9035802Z   test_multiple_local_shards (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63679
2022-11-23T02:31:22.9036343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63680
2022-11-23T02:31:22.9036770Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63681
2022-11-23T02:31:22.9037208Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63682
2022-11-23T02:31:22.9037811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9038243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9038875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9039343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9039918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9040344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9040906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9041370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9041922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9042362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9042921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9043388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9043945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9044385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9044946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9045405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9045817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9046282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9046742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9047190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9047571Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9048097Z   test_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63815
2022-11-23T02:31:22.9048632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63816
2022-11-23T02:31:22.9049059Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63817
2022-11-23T02:31:22.9049497Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63818
2022-11-23T02:31:22.9050101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9050528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9051095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9051568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9052141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9052570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9053136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9053596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9054147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9054587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9055148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9055664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9056218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9056659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9057213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9057671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9058312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9058791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9059257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9059704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9060096Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9060581Z   test_partial_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63951
2022-11-23T02:31:22.9061120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63952
2022-11-23T02:31:22.9061548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63953
2022-11-23T02:31:22.9061986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63954
2022-11-23T02:31:22.9062594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9063022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9063589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9064060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9064631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9065130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9065711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9066171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9066736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9067154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9067715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9068174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9068732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9069176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9069742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9070202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9070616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9071083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9071547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9071993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9072461Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9072952Z   test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64087
2022-11-23T02:31:22.9073500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64088
2022-11-23T02:31:22.9073929Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64089
2022-11-23T02:31:22.9074368Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64090
2022-11-23T02:31:22.9074974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9075402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9075968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9076436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9077016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9077441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9078009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9078473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9079044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9079465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9080025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9080485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9081043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9081479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9082094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9082557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9082972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9083437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9083897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9084346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9084736Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9085226Z   test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64223
2022-11-23T02:31:22.9085763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64224
2022-11-23T02:31:22.9086196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64225
2022-11-23T02:31:22.9086636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64226
2022-11-23T02:31:22.9087280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9087735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9088290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9088754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9089389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9089815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9090380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9090842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9091407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9091826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9092383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9092841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9093389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9093828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9094390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9094844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9095256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9095724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9096189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9096655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9097025Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9097508Z   test_sharding_columns (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64359
2022-11-23T02:31:22.9098230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64360
2022-11-23T02:31:22.9098746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64361
2022-11-23T02:31:22.9099194Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64362
2022-11-23T02:31:22.9099805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9100256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9100801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9101268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9101837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9102265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9102831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9103292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9103862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9104284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9104840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9105283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9105832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9106378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9106960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9107426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9107840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9108308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9108767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9109230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9109603Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:31:22.9110077Z   test_state_dict (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64495
2022-11-23T02:31:22.9110606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64496
2022-11-23T02:31:22.9111033Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64497
2022-11-23T02:31:22.9111473Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64498
2022-11-23T02:31:22.9112076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9112523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9113075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9113542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9114129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9114886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9115470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9116006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9116580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9117004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9117563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9118021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9118592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9119012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9119578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9120037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9120454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9120918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9121379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9121849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9122218Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9122682Z   test_state_dict_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64631
2022-11-23T02:31:22.9123274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64632
2022-11-23T02:31:22.9123704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64633
2022-11-23T02:31:22.9124153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64634
2022-11-23T02:31:22.9124743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9125176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9125724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9126187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9126753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9127176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9127743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9128203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9128766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9129188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9129746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9130205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9130764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9131183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9131752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9132210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9132673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9133149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9133606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9134073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9134445Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:31:22.9134945Z   test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64767
2022-11-23T02:31:22.9135494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64768
2022-11-23T02:31:22.9135927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64769
2022-11-23T02:31:22.9136371Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64770
2022-11-23T02:31:22.9136974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9137420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9138197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9138675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9139253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9139693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9140336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9140797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9141367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9141785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9142348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9142808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9143378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9143797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9144363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9144826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9145241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9145713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9146173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9146640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9147011Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9147485Z   test_custom_op (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64903
2022-11-23T02:31:22.9148015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64904
2022-11-23T02:31:22.9148466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64905
2022-11-23T02:31:22.9148891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64906
2022-11-23T02:31:22.9149561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9150016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9150571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9151034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9151604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9152045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9152592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9153059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9153626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9154052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9154614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9155075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9155643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9156066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9156630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9157148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9157580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9158030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9158494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9158957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9159325Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9159810Z   test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65039
2022-11-23T02:31:22.9160351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65040
2022-11-23T02:31:22.9160793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65041
2022-11-23T02:31:22.9161231Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65042
2022-11-23T02:31:22.9161833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9162281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9162829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9163295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9163863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9164302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9164849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9165315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9165879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9166351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9166920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9167377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9167947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9168365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9168924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9169385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9169820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9170266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9170721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9171187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9171556Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9172046Z   test_custom_op_override (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65175
2022-11-23T02:31:22.9172585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65176
2022-11-23T02:31:22.9173034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65177
2022-11-23T02:31:22.9173523Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65178
2022-11-23T02:31:22.9174121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9174568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9175121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9175588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9176163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9176604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9177152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9177611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9178350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9178792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9179344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9179800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9180371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9180793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9181357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9181814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9182247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9182698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9183224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9183699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9184071Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9184457Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable)
2022-11-23T02:31:22.9184951Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65311
2022-11-23T02:31:22.9185433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65312
2022-11-23T02:31:22.9185857Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65313
2022-11-23T02:31:22.9186300Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65314
2022-11-23T02:31:22.9186906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9187370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9187945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9188411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9188979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9189405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9189971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9190430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9191078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9191497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9192064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9192523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9193077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9193512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9194073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9194532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9194949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9195420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9195885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9196334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9196721Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9197085Z   test_gather_even (__main__.TestShardedTensorEnumerable)
2022-11-23T02:31:22.9197601Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65447
2022-11-23T02:31:22.9198115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65448
2022-11-23T02:31:22.9198560Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65449
2022-11-23T02:31:22.9199007Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65450
2022-11-23T02:31:22.9199597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9200090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9200668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9201136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9201690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9202134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9202695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9203163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9203712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9204155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9204712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9205155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9205730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9206170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9206730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9207173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9207665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9208135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9208583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9209045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9209430Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9209792Z   test_gather_uneven (__main__.TestShardedTensorEnumerable)
2022-11-23T02:31:22.9210293Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65583
2022-11-23T02:31:22.9210825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65584
2022-11-23T02:31:22.9211274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65585
2022-11-23T02:31:22.9211703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65586
2022-11-23T02:31:22.9212307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9212758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9213328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9213778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9214350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9214797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9215606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9216132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9216709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9217221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9217781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9218463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9219042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9219482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9220027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9220496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9220930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9221379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9221851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9222314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9222705Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9223173Z   test_grid_sharding (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65719
2022-11-23T02:31:22.9223716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65720
2022-11-23T02:31:22.9224166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65721
2022-11-23T02:31:22.9224707Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65722
2022-11-23T02:31:22.9225298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9225749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9226316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9226768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9227339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9227784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9228347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9228795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9229371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9229814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9230366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9230826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9231393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9231830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9232610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9233140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9233581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9234053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9234590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9235059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9235448Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9235926Z   test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65855
2022-11-23T02:31:22.9236475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65856
2022-11-23T02:31:22.9236926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65857
2022-11-23T02:31:22.9237370Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65858
2022-11-23T02:31:22.9237968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9238416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9238989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9239439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9240011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9240454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9241022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9241469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9242116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9242558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9243108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9243567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9244136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9244578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9245123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9245583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9246016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9246492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9246934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9247396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9247788Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:31:22.9248254Z   test_new_group (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65991
2022-11-23T02:31:22.9248788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65992
2022-11-23T02:31:22.9249239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65993
2022-11-23T02:31:22.9249679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65994
2022-11-23T02:31:22.9250271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9250716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9251353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9251812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9252179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9252353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9252725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9252913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9253273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9253447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9253819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9254002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9254344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9254517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9254890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9255077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9255304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9255585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9255801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9256028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9256160Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9256478Z   test_partial_world_size (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66127
2022-11-23T02:31:22.9256691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66128
2022-11-23T02:31:22.9256907Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66129
2022-11-23T02:31:22.9257122Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66130
2022-11-23T02:31:22.9257496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9257676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9258279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9258479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9258828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9258999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9259367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9259558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9259912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9260087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9260453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9260715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9261073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9261243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9261604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9261789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9262016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9262243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9262468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9262693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9262844Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9263157Z   test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66263
2022-11-23T02:31:22.9263374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66264
2022-11-23T02:31:22.9263587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66265
2022-11-23T02:31:22.9263802Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66266
2022-11-23T02:31:22.9264173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9264417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9264784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9264961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9265314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9265501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9265874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9266061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9266426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9266605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9266976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9267162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9267528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9267684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9268051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9268234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9268461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9268688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9268909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9269132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9269335Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9269651Z   test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66399
2022-11-23T02:31:22.9269866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66400
2022-11-23T02:31:22.9270077Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66401
2022-11-23T02:31:22.9270291Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66402
2022-11-23T02:31:22.9270659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9270837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9271213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9271402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9271764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9271918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9272284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9272472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9272825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9272995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9273421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9273607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9273972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9274126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9274492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9274674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9274902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9275126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9275342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9275572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9275721Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9276049Z   test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66535
2022-11-23T02:31:22.9276248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66536
2022-11-23T02:31:22.9276460Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66537
2022-11-23T02:31:22.9276674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66538
2022-11-23T02:31:22.9277044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9277218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9277596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9277785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9278192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9278352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9278714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9278886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9279256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9279445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9279814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9280007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9280374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9280546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9280893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9281077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9281303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9281528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9281746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9282025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9282174Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:31:22.9282505Z   test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66671
2022-11-23T02:31:22.9282702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66672
2022-11-23T02:31:22.9282917Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66673
2022-11-23T02:31:22.9283127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66674
2022-11-23T02:31:22.9283498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9283673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9284046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9284236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9284599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9284769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9285123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9285310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9285665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9285836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9286199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9286387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9286751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9286968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9287366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9287551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9287777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9288001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9288219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9288446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9288594Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9288921Z   test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66807
2022-11-23T02:31:22.9289138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66808
2022-11-23T02:31:22.9289334Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66809
2022-11-23T02:31:22.9289547Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66810
2022-11-23T02:31:22.9289924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9290097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9290470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9290713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9291077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9291253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9291601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9291789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9292146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9292318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9292686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9292875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9293239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9293413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9293778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9293944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9294172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9294397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9294617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9294839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9294989Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:31:22.9295305Z   test_uneven_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66943
2022-11-23T02:31:22.9295571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66944
2022-11-23T02:31:22.9295775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66945
2022-11-23T02:31:22.9295988Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66946
2022-11-23T02:31:22.9296358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9296533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9296909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9297102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9297461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9297633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9298219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9298396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9298763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9298935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9299299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9299484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9299964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9300134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9300503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9300670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9300898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9301123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9301343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9301566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9301718Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9302033Z   test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67079
2022-11-23T02:31:22.9302252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67080
2022-11-23T02:31:22.9302466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67081
2022-11-23T02:31:22.9302663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67082
2022-11-23T02:31:22.9303033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9303205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9303581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9303770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9304139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9304313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9304756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9304934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9305295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9305467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9305833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9306016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9306383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9306555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9306922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9307106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9307314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9307539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9307757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9307981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9308129Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:31:22.9309128Z   test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78068 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-11-23T02:31:22.9309496Z   test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67215
2022-11-23T02:31:22.9309711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67216
2022-11-23T02:31:22.9309924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67217
2022-11-23T02:31:22.9310137Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67218
2022-11-23T02:31:22.9310488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9310664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9311037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9311227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9311588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9311759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9312125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9312314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9312657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9312834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9313196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9313434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9313801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9313972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9314333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9314520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9314747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9314954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9315183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9315404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9315555Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9315938Z   test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67351
2022-11-23T02:31:22.9316157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67352
2022-11-23T02:31:22.9316368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67353
2022-11-23T02:31:22.9316579Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67354
2022-11-23T02:31:22.9316935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9317165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9317544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9317735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9318096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9318266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9318637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9318825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9319182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9319339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9319710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9319901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9320256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9320428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9320797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9320983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9321210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9321417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9321642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9321863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9322061Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:31:22.9322431Z   test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67487
2022-11-23T02:31:22.9322647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67488
2022-11-23T02:31:22.9322860Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67489
2022-11-23T02:31:22.9323071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67490
2022-11-23T02:31:22.9323442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9323602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9323974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9324163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9324516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9324697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9325070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9325259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9325617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9325841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9326194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9326376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9326739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9326908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9327281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9327467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9327690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9327910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9328113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9328338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9328486Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9328847Z   test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67623
2022-11-23T02:31:22.9329062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67624
2022-11-23T02:31:22.9329276Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67625
2022-11-23T02:31:22.9329487Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67626
2022-11-23T02:31:22.9329909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9330087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9330444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9330683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9331053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9331223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9331590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9331775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9332132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9332303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9332656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9332839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9333345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9333660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9334068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9334251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9334477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9334701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9334988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9335194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9335439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:22.9335680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:22.9335917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:31:22.9336151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:31:22.9336553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:31:22.9336948Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:31:22.9337340Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:31:22.9337737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:31:22.9337868Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:31:22.9338476Z   test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67771
2022-11-23T02:31:22.9338695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67772
2022-11-23T02:31:22.9338909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67773
2022-11-23T02:31:22.9339121Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67774
2022-11-23T02:31:22.9339494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9339672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9340047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9340314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9340668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9340839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9341206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9341389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9341751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9341925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9342293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9342482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9342824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9342996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9343356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9343540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9343767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9343985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9344276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9344494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9344644Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9344992Z   test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67907
2022-11-23T02:31:22.9345208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67908
2022-11-23T02:31:22.9345421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67909
2022-11-23T02:31:22.9345631Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67910
2022-11-23T02:31:22.9346005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9346183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9346558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9346750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9347094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9347264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9347631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9347815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9348176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9348351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9348719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9348953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9349321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9349475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9349849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9350035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9350261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9350487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9350713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9350927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9351076Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9351427Z   test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68043
2022-11-23T02:31:22.9351640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68044
2022-11-23T02:31:22.9351852Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68045
2022-11-23T02:31:22.9352064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68046
2022-11-23T02:31:22.9352433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9352661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9353041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9353231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9353589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9353740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9354111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9354293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9354658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9354831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9355199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9355381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9355739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9355912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9356262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9356441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9356666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9356885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9357112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9357329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9357526Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9357883Z   test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68179
2022-11-23T02:31:22.9358081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68180
2022-11-23T02:31:22.9358293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68181
2022-11-23T02:31:22.9358506Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68182
2022-11-23T02:31:22.9358883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9359062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9359435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9359625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9359985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9360157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9360508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9360691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9361057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9361227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9361651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9361836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9362197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9362368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9362718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9362901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9363125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9363349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9363574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9363791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9363939Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9364272Z   test_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68315
2022-11-23T02:31:22.9364485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68316
2022-11-23T02:31:22.9364682Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68317
2022-11-23T02:31:22.9364891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68318
2022-11-23T02:31:22.9365261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9365435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9365816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9366004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9366413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9366588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9366943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9367128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9367491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9367661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9368036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9368219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9368584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9368755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9369121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9369288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9369511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9369733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9369956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9370242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9370390Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9370764Z   test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68451
2022-11-23T02:31:22.9370980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68452
2022-11-23T02:31:22.9371176Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68453
2022-11-23T02:31:22.9371385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68454
2022-11-23T02:31:22.9371761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9371933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9372312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9372498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9372860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9373029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9373394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9373563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9373925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9374093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9374458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9374645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9375053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9375227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9375595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9375761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9375986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9376211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9376434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9376652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9376800Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:31:22.9377143Z   test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68587
2022-11-23T02:31:22.9377361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68588
2022-11-23T02:31:22.9377573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68589
2022-11-23T02:31:22.9377770Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68590
2022-11-23T02:31:22.9378357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9378534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9378910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9379177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9379542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9379711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9380083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9380251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9380614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9380785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9381153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9381338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9381698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9381873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9382240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9382424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9382631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9382855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9383076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9383297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9383444Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9383858Z   test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68723
2022-11-23T02:31:22.9384082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68724
2022-11-23T02:31:22.9384290Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68725
2022-11-23T02:31:22.9384486Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68726
2022-11-23T02:31:22.9384857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9385030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9385402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9385593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9385952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9386129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9386503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9386687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9387026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9387196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9387610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9387849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9388207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:22.9388381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:22.9388749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:22.9388931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:22.9389138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:22.9389360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:31:22.9389583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:31:22.9389801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:22.9389952Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:31:22.9390174Z   test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.055s)
2022-11-23T02:31:22.9390197Z 
2022-11-23T02:31:22.9390471Z ----------------------------------------------------------------------
2022-11-23T02:31:22.9390589Z Ran 64 tests in 149.247s
2022-11-23T02:31:22.9390608Z 
2022-11-23T02:31:22.9390717Z OK (skipped=62)
2022-11-23T02:31:22.9390737Z 
2022-11-23T02:31:22.9390844Z Generating XML reports...
2022-11-23T02:31:22.9391354Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221123022853.xml
2022-11-23T02:31:22.9391851Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221123022853.xml
2022-11-23T02:31:22.9392310Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221123022853.xml
2022-11-23T02:31:22.9392777Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221123022853.xml
2022-11-23T02:31:22.9393275Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221123022853.xml
2022-11-23T02:31:22.9393750Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221123022853.xml
2022-11-23T02:31:22.9394199Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221123022853.xml
2022-11-23T02:31:22.9394688Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221123022853.xml
2022-11-23T02:31:22.9395181Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221123022853.xml
2022-11-23T02:31:22.9395688Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221123022853.xml
2022-11-23T02:31:22.9396190Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221123022853.xml
2022-11-23T02:31:22.9396707Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221123022853.xml
2022-11-23T02:31:22.9396727Z 
2022-11-23T02:31:22.9397235Z ##[endgroup]
2022-11-23T02:31:22.9397778Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_igf6e9bi)
2022-11-23T02:31:22.9397861Z 
2022-11-23T02:31:22.9398159Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-11-23 02:31:22.873361]
2022-11-23T02:31:22.9398653Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:31:22.873753]
2022-11-23T02:34:59.4767103Z 
2022-11-23T02:34:59.4769791Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_summon_full_params
2022-11-23T02:34:59.4770779Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_sln702ni)
2022-11-23T02:34:59.4771173Z 
2022-11-23T02:34:59.4771291Z Running tests...
2022-11-23T02:34:59.4771804Z ----------------------------------------------------------------------
2022-11-23T02:34:59.4774485Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params
2022-11-23T02:34:59.4775088Z   test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:34:59.4776941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68894
2022-11-23T02:34:59.4777870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68895
2022-11-23T02:34:59.4779673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4780481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4781075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4781678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4782645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4783526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4784540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4785058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4785942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4786813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4788177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4789443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4790377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4791238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4793491Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.4794977Z   warnings.warn(
2022-11-23T02:34:59.4796999Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.4798563Z   warnings.warn(
2022-11-23T02:34:59.4799001Z dist init r=1, world=2
2022-11-23T02:34:59.4799443Z dist init r=0, world=2
2022-11-23T02:34:59.4799809Z ok (6.083s)
2022-11-23T02:34:59.4800577Z   test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68977
2022-11-23T02:34:59.4801501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68978
2022-11-23T02:34:59.4803117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4803857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4804791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4805584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4806564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4807304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4808269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4808760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4809217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4809696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4810365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4811060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4811593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4812048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4813391Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.4814201Z   warnings.warn(
2022-11-23T02:34:59.4815364Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.4816149Z   warnings.warn(
2022-11-23T02:34:59.4816382Z dist init r=0, world=2
2022-11-23T02:34:59.4816634Z dist init r=1, world=2
2022-11-23T02:34:59.4816875Z ok (3.911s)
2022-11-23T02:34:59.4817212Z   test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams)
2022-11-23T02:34:59.4817755Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69056
2022-11-23T02:34:59.4818613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69057
2022-11-23T02:34:59.4819236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4819787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4820365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4820845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4821430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4821861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4822436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4822903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4823335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4823839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4824502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4825189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4825697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4826171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4826534Z dist init r=0, world=2
2022-11-23T02:34:59.4826771Z dist init r=1, world=2
2022-11-23T02:34:59.4827012Z ok (3.911s)
2022-11-23T02:34:59.4827366Z   test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams)
2022-11-23T02:34:59.4827909Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69135
2022-11-23T02:34:59.4828423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69136
2022-11-23T02:34:59.4829037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4829561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4830136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4830612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4831197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4831646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4832204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4832681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4833135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4833612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4834271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4834956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4835477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4835933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4836294Z dist init r=0, world=2
2022-11-23T02:34:59.4836549Z dist init r=1, world=2
2022-11-23T02:34:59.4836832Z ok (3.911s)
2022-11-23T02:34:59.4837206Z   test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams)
2022-11-23T02:34:59.4837766Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69214
2022-11-23T02:34:59.4838290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69215
2022-11-23T02:34:59.4838882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4839334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4839914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4840381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4840944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4841397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4841967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4842415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4842865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4843361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4844015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4844692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4845220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4845697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4846053Z dist init r=0, world=2
2022-11-23T02:34:59.4846289Z dist init r=1, world=2
2022-11-23T02:34:59.4846583Z ok (3.911s)
2022-11-23T02:34:59.4846968Z   test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams)
2022-11-23T02:34:59.4847510Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69293
2022-11-23T02:34:59.4848035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69294
2022-11-23T02:34:59.4848654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4849088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4849678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4850147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4850733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4851168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4851745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4852212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4852669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4853148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4853817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4854569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4855077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4855553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4855908Z dist init r=1, world=2
2022-11-23T02:34:59.4856165Z dist init r=0, world=2
2022-11-23T02:34:59.4856386Z ok (3.911s)
2022-11-23T02:34:59.4856903Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69372
2022-11-23T02:34:59.4857515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69373
2022-11-23T02:34:59.4858465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4858927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4859511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4859982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4860538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4860992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4861564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4862031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4862460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4862958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4863616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4864359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4864900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4865368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4865730Z dist init r=0, world=2
2022-11-23T02:34:59.4865966Z dist init r=1, world=2
2022-11-23T02:34:59.4866209Z ok (3.912s)
2022-11-23T02:34:59.4866727Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69451
2022-11-23T02:34:59.4867322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69452
2022-11-23T02:34:59.4867938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4868389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4868964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4869418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4869998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4870444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4870995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4871596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4872044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4872540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4873177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4873865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4874386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4874860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4875196Z dist init r=1, world=2
2022-11-23T02:34:59.4875449Z dist init r=0, world=2
2022-11-23T02:34:59.4875695Z ok (4.011s)
2022-11-23T02:34:59.4876193Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69530
2022-11-23T02:34:59.4876802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69531
2022-11-23T02:34:59.4877414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4877866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4878426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4878899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4879478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4879911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4880482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4880992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4881451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4881927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4882581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4883274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4883798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4884261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4885345Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.4886029Z   warnings.warn(
2022-11-23T02:34:59.4887041Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.4887698Z   warnings.warn(
2022-11-23T02:34:59.4888739Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.4889401Z   warnings.warn(
2022-11-23T02:34:59.4890354Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.4891007Z   warnings.warn(
2022-11-23T02:34:59.4891239Z dist init r=0, world=2
2022-11-23T02:34:59.4891490Z dist init r=1, world=2
2022-11-23T02:34:59.4891736Z ok (3.913s)
2022-11-23T02:34:59.4892228Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69609
2022-11-23T02:34:59.4892842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69610
2022-11-23T02:34:59.4893456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4893911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4894464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4894930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4895509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4895937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4896510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4896978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4897480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4898249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4898924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4899619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4900141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4900596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4901679Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.4902359Z   warnings.warn(
2022-11-23T02:34:59.4903325Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.4903997Z   warnings.warn(
2022-11-23T02:34:59.4904935Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.4905686Z   warnings.warn(
2022-11-23T02:34:59.4906640Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.4907296Z   warnings.warn(
2022-11-23T02:34:59.4907527Z dist init r=1, world=2
2022-11-23T02:34:59.4907779Z dist init r=0, world=2
2022-11-23T02:34:59.4908020Z ok (3.911s)
2022-11-23T02:34:59.4908512Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69688
2022-11-23T02:34:59.4909128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69689
2022-11-23T02:34:59.4909741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4910196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4910751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4911219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4911796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4912240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4912792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4913256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4913706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4914250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4914917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4915608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4916135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4916587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4916934Z dist init r=1, world=2
2022-11-23T02:34:59.4917189Z dist init r=0, world=2
2022-11-23T02:34:59.4917413Z ok (4.011s)
2022-11-23T02:34:59.4917924Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69767
2022-11-23T02:34:59.4918528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69768
2022-11-23T02:34:59.4919142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4919577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4920148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4920618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4921180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4921686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4922262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4922737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4923171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4923666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4924320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4925011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4925515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4925992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4926351Z dist init r=1, world=2
2022-11-23T02:34:59.4926588Z dist init r=0, world=2
2022-11-23T02:34:59.4926831Z ok (3.912s)
2022-11-23T02:34:59.4927346Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69846
2022-11-23T02:34:59.4927954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69847
2022-11-23T02:34:59.4928546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4929000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4929576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4930033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4930617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4931114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4931698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4932144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4932600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4933097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4933748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4934429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4934952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4935428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4935770Z dist init r=1, world=2
2022-11-23T02:34:59.4936025Z dist init r=0, world=2
2022-11-23T02:34:59.4936266Z ok (4.012s)
2022-11-23T02:34:59.4936758Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69925
2022-11-23T02:34:59.4937362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69926
2022-11-23T02:34:59.4938281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4938829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4939376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4939829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4940403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4940876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4941444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4941910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4942366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4942839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4943496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4944188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4944712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4945170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4945527Z dist init r=1, world=2
2022-11-23T02:34:59.4945783Z dist init r=0, world=2
2022-11-23T02:34:59.4946006Z ok (3.912s)
2022-11-23T02:34:59.4946521Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70004
2022-11-23T02:34:59.4947126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70005
2022-11-23T02:34:59.4947738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4948234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4948824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4949296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4949871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4950299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4950867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4951336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4951771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4952264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4952918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4953611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4954117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4954586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4954945Z dist init r=0, world=2
2022-11-23T02:34:59.4955196Z dist init r=1, world=2
2022-11-23T02:34:59.4955422Z ok (3.912s)
2022-11-23T02:34:59.4955930Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70083
2022-11-23T02:34:59.4956594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70084
2022-11-23T02:34:59.4957187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4957640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4958210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4958679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4959237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4959678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4960257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4960705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4961158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4961656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4962310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4962981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4963501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4964068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4964594Z dist init r=0, world=2
2022-11-23T02:34:59.4964872Z dist init r=1, world=2
2022-11-23T02:34:59.4965186Z ok (3.911s)
2022-11-23T02:34:59.4965823Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70162
2022-11-23T02:34:59.4966457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70163
2022-11-23T02:34:59.4967206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4967735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4968382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4985181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4985846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4986282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4986910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4987380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4987815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4988306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4988961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4989644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4990309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.4990780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.4991135Z dist init r=0, world=2
2022-11-23T02:34:59.4991374Z dist init r=1, world=2
2022-11-23T02:34:59.4991612Z ok (3.911s)
2022-11-23T02:34:59.4992120Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70241
2022-11-23T02:34:59.4992712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70242
2022-11-23T02:34:59.4993307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4993755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4994332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4994798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4995355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.4995793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.4996356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.4996796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.4997244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.4997738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.4998388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4999054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.4999643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5000124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5000461Z dist init r=1, world=2
2022-11-23T02:34:59.5000711Z dist init r=0, world=2
2022-11-23T02:34:59.5000949Z ok (4.011s)
2022-11-23T02:34:59.5001457Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70320
2022-11-23T02:34:59.5002036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70321
2022-11-23T02:34:59.5002644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5003097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5003653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5004119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5004693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5005132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5005678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5006137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5006580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5007150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5007788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5008471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5008985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5009431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5009780Z dist init r=0, world=2
2022-11-23T02:34:59.5010030Z dist init r=1, world=2
2022-11-23T02:34:59.5010264Z ok (3.911s)
2022-11-23T02:34:59.5010755Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70399
2022-11-23T02:34:59.5011355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70400
2022-11-23T02:34:59.5011959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5012389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5012960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5013420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5013991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5014410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5014971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5015431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5015861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5016408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5017064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5017750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5018670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5019134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5019480Z dist init r=0, world=2
2022-11-23T02:34:59.5019728Z dist init r=1, world=2
2022-11-23T02:34:59.5019954Z ok (3.911s)
2022-11-23T02:34:59.5020460Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70478
2022-11-23T02:34:59.5021057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70479
2022-11-23T02:34:59.5021656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5022102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5022675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5023137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5023690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5024231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5024799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5025242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5025690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5026179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5026827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5027496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5028014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5028486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5028834Z dist init r=0, world=2
2022-11-23T02:34:59.5029066Z dist init r=1, world=2
2022-11-23T02:34:59.5029302Z ok (3.911s)
2022-11-23T02:34:59.5029809Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70557
2022-11-23T02:34:59.5030385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70558
2022-11-23T02:34:59.5030990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5031434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5032006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5032455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5033024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5033457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5034076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5034547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5034989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5035479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5036112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5036796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5037318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5037784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5038116Z dist init r=1, world=2
2022-11-23T02:34:59.5038363Z dist init r=0, world=2
2022-11-23T02:34:59.5038598Z ok (3.911s)
2022-11-23T02:34:59.5038896Z   test_raises_rank0_with_writeback (__main__.TestSummonFullParams)
2022-11-23T02:34:59.5039398Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70636
2022-11-23T02:34:59.5039910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70637
2022-11-23T02:34:59.5040499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5041002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5041572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5042039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5042593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5043031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5043595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5044053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5044477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5044972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5045626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5046293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5046804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5047270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5047616Z dist init r=1, world=2
2022-11-23T02:34:59.5047847Z dist init r=0, world=2
2022-11-23T02:34:59.5048084Z ok (3.910s)
2022-11-23T02:34:59.5048622Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70715
2022-11-23T02:34:59.5049240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70716
2022-11-23T02:34:59.5049851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5050347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5050926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5051372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5051938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5052374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5052941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5053381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5053831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5054321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5054957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5055639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5056152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5056614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5057859Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5058998Z   warnings.warn(
2022-11-23T02:34:59.5060151Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5060917Z   warnings.warn(
2022-11-23T02:34:59.5061688Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5062247Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5063047Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5063617Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5063895Z dist init r=1, world=2
2022-11-23T02:34:59.5064124Z dist init r=0, world=2
2022-11-23T02:34:59.5064358Z ok (4.413s)
2022-11-23T02:34:59.5064893Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70798
2022-11-23T02:34:59.5065513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70799
2022-11-23T02:34:59.5066114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5066558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5067201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5067659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5068237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5068674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5069241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5069681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5070132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5070624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5071262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5071947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5072461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5072923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5074183Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5075008Z   warnings.warn(
2022-11-23T02:34:59.5076221Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5076976Z   warnings.warn(
2022-11-23T02:34:59.5077743Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5078324Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5079112Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5079682Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5079961Z dist init r=1, world=2
2022-11-23T02:34:59.5080192Z dist init r=0, world=2
2022-11-23T02:34:59.5080429Z ok (4.413s)
2022-11-23T02:34:59.5080964Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70881
2022-11-23T02:34:59.5081583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70882
2022-11-23T02:34:59.5082175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5082619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5083238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5083711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5084266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5084702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5085264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5085702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5086158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5086692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5087355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5088017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5088531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5088998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5090259Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5091074Z   warnings.warn(
2022-11-23T02:34:59.5092230Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5092981Z   warnings.warn(
2022-11-23T02:34:59.5093745Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5094319Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5095101Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5095668Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5096674Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5097329Z   warnings.warn(
2022-11-23T02:34:59.5098535Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5099249Z   warnings.warn(
2022-11-23T02:34:59.5100235Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5100887Z   warnings.warn(
2022-11-23T02:34:59.5101841Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5102489Z   warnings.warn(
2022-11-23T02:34:59.5102716Z dist init r=1, world=2
2022-11-23T02:34:59.5102963Z dist init r=0, world=2
2022-11-23T02:34:59.5103202Z ok (4.413s)
2022-11-23T02:34:59.5103719Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70964
2022-11-23T02:34:59.5104338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70965
2022-11-23T02:34:59.5104949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5105375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5105947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5106487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5107066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5107488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5108050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5108506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5108954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5109430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5110083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5110776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5111280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5111748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5113008Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5113779Z   warnings.warn(
2022-11-23T02:34:59.5114971Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5115730Z   warnings.warn(
2022-11-23T02:34:59.5116491Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5117071Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5117866Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5118437Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5119426Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5120081Z   warnings.warn(
2022-11-23T02:34:59.5121036Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5121740Z   warnings.warn(
2022-11-23T02:34:59.5122720Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5123360Z   warnings.warn(
2022-11-23T02:34:59.5124316Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5124958Z   warnings.warn(
2022-11-23T02:34:59.5125204Z dist init r=0, world=2
2022-11-23T02:34:59.5125435Z dist init r=1, world=2
2022-11-23T02:34:59.5125670Z ok (4.413s)
2022-11-23T02:34:59.5126207Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71047
2022-11-23T02:34:59.5126817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71048
2022-11-23T02:34:59.5127426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5127869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5128439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5128888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5129462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5129906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5130473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5130965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5131421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5131917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5132554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5133242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5133756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5134232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5135476Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5136238Z   warnings.warn(
2022-11-23T02:34:59.5137393Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5138456Z   warnings.warn(
2022-11-23T02:34:59.5139242Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5139799Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5140602Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5141176Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5141460Z dist init r=1, world=2
2022-11-23T02:34:59.5141687Z dist init r=0, world=2
2022-11-23T02:34:59.5141921Z ok (4.413s)
2022-11-23T02:34:59.5142461Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71130
2022-11-23T02:34:59.5143086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71131
2022-11-23T02:34:59.5143679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5144123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5144693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5145139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5145710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5146147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5146720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5147161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5147681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5148185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5148825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5149509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5150027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5150486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5151756Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5152513Z   warnings.warn(
2022-11-23T02:34:59.5153655Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5154500Z   warnings.warn(
2022-11-23T02:34:59.5155278Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5155851Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5156635Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5157208Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5157486Z dist init r=0, world=2
2022-11-23T02:34:59.5157719Z dist init r=1, world=2
2022-11-23T02:34:59.5157954Z ok (4.413s)
2022-11-23T02:34:59.5158484Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71213
2022-11-23T02:34:59.5159109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71214
2022-11-23T02:34:59.5159704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5160146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5160717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5161188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5161744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5162182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5162751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5163195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5163694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5164194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5164847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5165516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5166027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5166493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5167761Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5168506Z   warnings.warn(
2022-11-23T02:34:59.5169644Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5170472Z   warnings.warn(
2022-11-23T02:34:59.5171247Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5171823Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5172606Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5173175Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5173452Z dist init r=1, world=2
2022-11-23T02:34:59.5173702Z dist init r=0, world=2
2022-11-23T02:34:59.5173920Z ok (4.513s)
2022-11-23T02:34:59.5174448Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71296
2022-11-23T02:34:59.5175063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71297
2022-11-23T02:34:59.5175658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5176103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5176673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5177132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5177687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5178361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5178941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5179380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5179902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5180407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5181060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5181722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5182240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5182706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5183973Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5184746Z   warnings.warn(
2022-11-23T02:34:59.5185874Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5186739Z   warnings.warn(
2022-11-23T02:34:59.5187516Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5188092Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5188867Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5189435Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5189712Z dist init r=1, world=2
2022-11-23T02:34:59.5189957Z dist init r=0, world=2
2022-11-23T02:34:59.5190176Z ok (4.414s)
2022-11-23T02:34:59.5190601Z   test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71379
2022-11-23T02:34:59.5191126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71380
2022-11-23T02:34:59.5191714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5192161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5192733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5193195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5193749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5194188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5194751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5195200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5195646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5196193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5196854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5197519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5198034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5198500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5198849Z dist init r=0, world=2
2022-11-23T02:34:59.5199080Z dist init r=1, world=2
2022-11-23T02:34:59.5199320Z ok (3.911s)
2022-11-23T02:34:59.5199831Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71458
2022-11-23T02:34:59.5200415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71459
2022-11-23T02:34:59.5201015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5201456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5202032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5202479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5203047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5203541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5204098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5204563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5205014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5205505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5206135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5206823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5207335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5207804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5209050Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5209829Z   warnings.warn(
2022-11-23T02:34:59.5210976Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5211731Z   warnings.warn(
2022-11-23T02:34:59.5211979Z dist init r=1, world=2
2022-11-23T02:34:59.5212208Z dist init r=0, world=2
2022-11-23T02:34:59.5212441Z ok (4.012s)
2022-11-23T02:34:59.5213012Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71537
2022-11-23T02:34:59.5213600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71538
2022-11-23T02:34:59.5214207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5214653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5215220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5215671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5216244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5216682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5217233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5217695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5218399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5218895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5219535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5220305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5220817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5221290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5222526Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5223302Z   warnings.warn(
2022-11-23T02:34:59.5224449Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5225217Z   warnings.warn(
2022-11-23T02:34:59.5225465Z dist init r=1, world=2
2022-11-23T02:34:59.5225694Z dist init r=0, world=2
2022-11-23T02:34:59.5225931Z ok (3.911s)
2022-11-23T02:34:59.5226433Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71616
2022-11-23T02:34:59.5227011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71617
2022-11-23T02:34:59.5227619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5228067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5228635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5229143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5229729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5230166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5230715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5231172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5231617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5232113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5232746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5233431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5233949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5234411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5235651Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5236479Z   warnings.warn(
2022-11-23T02:34:59.5237623Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5238388Z   warnings.warn(
2022-11-23T02:34:59.5238636Z dist init r=0, world=2
2022-11-23T02:34:59.5238864Z dist init r=1, world=2
2022-11-23T02:34:59.5239098Z ok (4.012s)
2022-11-23T02:34:59.5239604Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71695
2022-11-23T02:34:59.5240184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71696
2022-11-23T02:34:59.5240793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5241243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5241810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5242256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5242825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5243257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5243819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5244264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5244707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5245248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5245891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5246570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5247083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5247550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5248790Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5249718Z   warnings.warn(
2022-11-23T02:34:59.5250872Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5251641Z   warnings.warn(
2022-11-23T02:34:59.5251887Z dist init r=1, world=2
2022-11-23T02:34:59.5252179Z dist init r=0, world=2
2022-11-23T02:34:59.5252417Z ok (3.911s)
2022-11-23T02:34:59.5252931Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71774
2022-11-23T02:34:59.5253532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71775
2022-11-23T02:34:59.5254126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5254571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5255139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5255587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5256154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5256599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5257165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5257607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5258277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5258779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5259420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5260101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5260617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5261089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5262399Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5263186Z   warnings.warn(
2022-11-23T02:34:59.5264332Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5265089Z   warnings.warn(
2022-11-23T02:34:59.5265338Z dist init r=1, world=2
2022-11-23T02:34:59.5265571Z dist init r=0, world=2
2022-11-23T02:34:59.5265805Z ok (3.911s)
2022-11-23T02:34:59.5266312Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71853
2022-11-23T02:34:59.5266906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71854
2022-11-23T02:34:59.5267498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5267941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5268499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5269002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5269572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5270037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5270615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5271057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5271499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5271987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5272622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5273315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5273834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5274298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5275551Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5276304Z   warnings.warn(
2022-11-23T02:34:59.5277495Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5278279Z   warnings.warn(
2022-11-23T02:34:59.5278526Z dist init r=1, world=2
2022-11-23T02:34:59.5278756Z dist init r=0, world=2
2022-11-23T02:34:59.5278987Z ok (4.011s)
2022-11-23T02:34:59.5279495Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71932
2022-11-23T02:34:59.5280089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71933
2022-11-23T02:34:59.5280681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5281129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5281702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5282152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5282724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5283164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5283731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5284174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5284620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5285170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5285832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5286499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5287055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5287524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5288785Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5289533Z   warnings.warn(
2022-11-23T02:34:59.5290667Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5291437Z   warnings.warn(
2022-11-23T02:34:59.5291684Z dist init r=1, world=2
2022-11-23T02:34:59.5291913Z dist init r=0, world=2
2022-11-23T02:34:59.5292146Z ok (4.011s)
2022-11-23T02:34:59.5292650Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72011
2022-11-23T02:34:59.5293242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72012
2022-11-23T02:34:59.5293881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5294335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5294909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5295353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5295922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5296360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5296923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5297366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5297809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5298552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5299213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5299880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5300394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5300859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5302122Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5302954Z   warnings.warn(
2022-11-23T02:34:59.5304101Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5304869Z   warnings.warn(
2022-11-23T02:34:59.5305119Z dist init r=0, world=2
2022-11-23T02:34:59.5305350Z dist init r=1, world=2
2022-11-23T02:34:59.5305584Z ok (4.011s)
2022-11-23T02:34:59.5306059Z   test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72090
2022-11-23T02:34:59.5306621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72091
2022-11-23T02:34:59.5307210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5307655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5308228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5308675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5309252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5309701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5310328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5310779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5311224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5311717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5312373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5313037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5313557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5314019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5314353Z dist init r=0, world=2
2022-11-23T02:34:59.5314603Z dist init r=1, world=2
2022-11-23T02:34:59.5314844Z ok (3.911s)
2022-11-23T02:34:59.5315298Z   test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72169
2022-11-23T02:34:59.5315857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72170
2022-11-23T02:34:59.5316461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5316906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5317457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5317983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5318558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5318999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5319548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5320000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5320444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5320918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5321571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5322253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5322764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5323213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5323556Z dist init r=1, world=2
2022-11-23T02:34:59.5323806Z dist init r=0, world=2
2022-11-23T02:34:59.5324026Z ok (4.011s)
2022-11-23T02:34:59.5324466Z   test_summon_full_param_writeback (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72248
2022-11-23T02:34:59.5324995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72249
2022-11-23T02:34:59.5325600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5326025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5326599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5327060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5327672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5328119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5328688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5329145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5329574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5330067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5330720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5331404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5331900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5332365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5332714Z dist init r=0, world=2
2022-11-23T02:34:59.5332947Z dist init r=1, world=2
2022-11-23T02:34:59.5333184Z ok (4.010s)
2022-11-23T02:34:59.5333674Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72327
2022-11-23T02:34:59.5334248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72328
2022-11-23T02:34:59.5334898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5335340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5335916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5336364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5336938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5337375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5338157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5338611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5339062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5339553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5340210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5340882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5341396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5341854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5342189Z dist init r=1, world=2
2022-11-23T02:34:59.5342434Z dist init r=0, world=2
2022-11-23T02:34:59.5342672Z ok (3.911s)
2022-11-23T02:34:59.5343144Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72406
2022-11-23T02:34:59.5343724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72407
2022-11-23T02:34:59.5344400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5344856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5345414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5345881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5346456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5346894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5347441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5347903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5348346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5348825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5349479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5350157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5350667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5351113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5352194Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5352947Z   warnings.warn(
2022-11-23T02:34:59.5353906Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5354546Z   warnings.warn(
2022-11-23T02:34:59.5355498Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5356152Z   warnings.warn(
2022-11-23T02:34:59.5357102Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-11-23T02:34:59.5357744Z   warnings.warn(
2022-11-23T02:34:59.5357973Z dist init r=0, world=2
2022-11-23T02:34:59.5358218Z dist init r=1, world=2
2022-11-23T02:34:59.5358453Z ok (3.911s)
2022-11-23T02:34:59.5358946Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72485
2022-11-23T02:34:59.5359510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72486
2022-11-23T02:34:59.5360108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5360603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5361168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5361629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5362201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5362638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5363189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5363649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5364095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5364592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5365229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5365911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5366427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5366655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5366751Z dist init r=0, world=2
2022-11-23T02:34:59.5366862Z dist init r=1, world=2
2022-11-23T02:34:59.5366962Z ok (3.911s)
2022-11-23T02:34:59.5367383Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72564
2022-11-23T02:34:59.5367603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72565
2022-11-23T02:34:59.5367978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5368153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5368530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5368703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5369066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5369237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5369614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5369799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5370046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5370284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5370680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5371068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5371278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5371503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5371619Z dist init r=1, world=2
2022-11-23T02:34:59.5371728Z dist init r=0, world=2
2022-11-23T02:34:59.5371827Z ok (3.911s)
2022-11-23T02:34:59.5372219Z   test_summon_full_params_respects_reshard_after_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72643
2022-11-23T02:34:59.5372445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72644
2022-11-23T02:34:59.5372817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5372973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5373352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5373541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5373908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5374079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5374460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5374648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5374886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5375126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5375510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5375954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5376234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5376462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5377493Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5377609Z   warnings.warn(
2022-11-23T02:34:59.5378824Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5378943Z   warnings.warn(
2022-11-23T02:34:59.5379582Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5379725Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5380353Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:34:59.5380497Z   warnings.warn(message, UserWarning)
2022-11-23T02:34:59.5380596Z dist init r=0, world=2
2022-11-23T02:34:59.5380705Z dist init r=1, world=2
2022-11-23T02:34:59.5380805Z ok (4.411s)
2022-11-23T02:34:59.5381112Z   test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72722
2022-11-23T02:34:59.5381403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72723
2022-11-23T02:34:59.5381791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5381967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5382346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5382519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5382883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5383060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5383433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5383621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5383867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5384110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5384511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5384904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5385112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5385407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5386431Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5386547Z   warnings.warn(
2022-11-23T02:34:59.5387578Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:34:59.5387696Z   warnings.warn(
2022-11-23T02:34:59.5387808Z dist init r=0, world=2
2022-11-23T02:34:59.5387919Z dist init r=1, world=2
2022-11-23T02:34:59.5388023Z ok (3.911s)
2022-11-23T02:34:59.5388199Z   test_with_grads_core (__main__.TestSummonFullParams)
2022-11-23T02:34:59.5388486Z Tests the core usage of ``summon_full_params(with_grads=True)``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72801
2022-11-23T02:34:59.5388705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72802
2022-11-23T02:34:59.5389084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5389260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5389637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5389832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5390194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5390414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5390778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5390965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5391207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5391451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5391852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5392253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5392480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5392707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5392943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5393159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5393383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5393612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5393843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5394124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5394346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5394572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5394798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5395024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5395232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5395456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5395676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5395900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5396130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5396354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5396578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5396799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5397004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5397229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5397451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5397672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5397892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5398118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:34:59.5398230Z dist init r=0, world=2
2022-11-23T02:34:59.5398340Z dist init r=1, world=2
2022-11-23T02:34:59.5398422Z ok (6.915s)
2022-11-23T02:34:59.5398651Z   test_with_grads_none_grads (__main__.TestSummonFullParams)
2022-11-23T02:34:59.5399110Z Tests that if all ranks' ``FlatParameter`` has ``None`` gradient, then ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72884
2022-11-23T02:34:59.5399329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72885
2022-11-23T02:34:59.5399703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5399876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5400252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5400447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5400793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5400970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5401344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5401531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5401775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:34:59.5402015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5402413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5402863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:34:59.5403091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:34:59.5403301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5403414Z dist init r=0, world=2
2022-11-23T02:34:59.5403523Z dist init r=1, world=2
2022-11-23T02:34:59.5403623Z ok (4.010s)
2022-11-23T02:34:59.5403955Z   test_summon_full_param_writeback (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72963
2022-11-23T02:34:59.5404330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:34:59.5404505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:34:59.5404883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:34:59.5405053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:34:59.5405298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:34:59.5405691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:34:59.5405918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:34:59.5406029Z dist init r=0, world=1
2022-11-23T02:34:59.5406131Z ok (3.809s)
2022-11-23T02:34:59.5406153Z 
2022-11-23T02:34:59.5406423Z ----------------------------------------------------------------------
2022-11-23T02:34:59.5406541Z Ran 52 tests in 214.278s
2022-11-23T02:34:59.5406561Z 
2022-11-23T02:34:59.5406637Z OK
2022-11-23T02:34:59.5406656Z 
2022-11-23T02:34:59.5406787Z Generating XML reports...
2022-11-23T02:34:59.5407263Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221123023124.xml
2022-11-23T02:34:59.5407810Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221123023124.xml
2022-11-23T02:34:59.5407832Z 
2022-11-23T02:34:59.5408275Z ##[endgroup]
2022-11-23T02:34:59.5408794Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_sln702ni)
2022-11-23T02:34:59.5408815Z 
2022-11-23T02:34:59.5409091Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-11-23 02:34:59.478098]
2022-11-23T02:34:59.5409575Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:34:59.478359]
2022-11-23T02:42:53.8222365Z 
2022-11-23T02:42:53.8223114Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict
2022-11-23T02:42:53.8224296Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict__e0_b_k9)
2022-11-23T02:42:53.8226993Z 
2022-11-23T02:42:53.8227394Z Running tests...
2022-11-23T02:42:53.8228202Z ----------------------------------------------------------------------
2022-11-23T02:42:53.8228996Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict
2022-11-23T02:42:53.8229960Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8230619Z Tests that we can save a state_dict and load it into a blank model ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:42:53.8231423Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73038
2022-11-23T02:42:53.8231879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73039
2022-11-23T02:42:53.8232529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8232978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8233547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8234022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8234603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8236587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8237467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8237970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8238429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8238910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8239570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8240260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8240772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8241218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8241564Z dist init r=0, world=2
2022-11-23T02:42:53.8246023Z dist init r=1, world=2
2022-11-23T02:42:53.8246486Z ok (5.620s)
2022-11-23T02:42:53.8247463Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8249038Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73117
2022-11-23T02:42:53.8250046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73118
2022-11-23T02:42:53.8251167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8251945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8252973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8253979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8255071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8255871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8256998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8257866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8259290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8260248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8261607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8262970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8264195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8265167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8265849Z dist init r=0, world=2
2022-11-23T02:42:53.8266624Z dist init r=1, world=2
2022-11-23T02:42:53.8267067Z ok (3.912s)
2022-11-23T02:42:53.8268048Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8269432Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73196
2022-11-23T02:42:53.8270478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73197
2022-11-23T02:42:53.8271722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8272612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8273763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8274705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8275883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8276764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8277939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8278879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8279779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8280756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8282274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8283703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8284828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8285768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8286484Z dist init r=0, world=2
2022-11-23T02:42:53.8286949Z dist init r=1, world=2
2022-11-23T02:42:53.8287413Z ok (3.912s)
2022-11-23T02:42:53.8288400Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8289791Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73275
2022-11-23T02:42:53.8290855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73276
2022-11-23T02:42:53.8292101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8292988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8294190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8295110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8296261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8297303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8298848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8299817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8300700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8301682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8302996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8304400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8305413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8306360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8307083Z dist init r=0, world=2
2022-11-23T02:42:53.8307548Z dist init r=1, world=2
2022-11-23T02:42:53.8307982Z ok (3.912s)
2022-11-23T02:42:53.8308996Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8310396Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73354
2022-11-23T02:42:53.8311412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73355
2022-11-23T02:42:53.8312683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8313571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8314702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8315618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8316951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8317862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8319018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8319979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8320888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8321861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8323185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8324578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8325591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8326539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8327215Z dist init r=0, world=2
2022-11-23T02:42:53.8327698Z dist init r=1, world=2
2022-11-23T02:42:53.8328155Z ok (4.012s)
2022-11-23T02:42:53.8329087Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8330412Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73433
2022-11-23T02:42:53.8331637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73434
2022-11-23T02:42:53.8332889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8333763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8334938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8335880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8337026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8338270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8339463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8340393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8341293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8342301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8343658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8345080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8346083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8347030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8347737Z dist init r=0, world=2
2022-11-23T02:42:53.8348189Z dist init r=1, world=2
2022-11-23T02:42:53.8348633Z ok (3.811s)
2022-11-23T02:42:53.8349631Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8351207Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73512
2022-11-23T02:42:53.8352259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73513
2022-11-23T02:42:53.8353492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8354381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8355525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8356481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8357671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8358565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8359817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8360755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8361654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8362639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8363976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8365366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8366578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8367518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8368247Z dist init r=0, world=2
2022-11-23T02:42:53.8368716Z dist init r=1, world=2
2022-11-23T02:42:53.8369139Z ok (3.911s)
2022-11-23T02:42:53.8370151Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8371520Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73591
2022-11-23T02:42:53.8372579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73592
2022-11-23T02:42:53.8373818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8374721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8375888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8376813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8378380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8379287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8380471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8381397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8382258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8383246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8384809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8386173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8387214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8388151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8388843Z dist init r=0, world=2
2022-11-23T02:42:53.8389304Z dist init r=1, world=2
2022-11-23T02:42:53.8389773Z ok (3.911s)
2022-11-23T02:42:53.8390746Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8392116Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73670
2022-11-23T02:42:53.8393189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73671
2022-11-23T02:42:53.8394444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8395337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8396478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8397384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8398562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8399636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8400792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8401744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8402625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8403593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8404941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8406366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8407428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8408368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8409068Z dist init r=1, world=2
2022-11-23T02:42:53.8409550Z dist init r=0, world=2
2022-11-23T02:42:53.8409996Z ok (4.012s)
2022-11-23T02:42:53.8410987Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8412357Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73749
2022-11-23T02:42:53.8413425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73750
2022-11-23T02:42:53.8414641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8415532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8416717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8417659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8419340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8420244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8421426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8422340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8423237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8424221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8425569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8426983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8428026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8428961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8429642Z dist init r=0, world=2
2022-11-23T02:42:53.8430069Z dist init r=1, world=2
2022-11-23T02:42:53.8430522Z ok (3.812s)
2022-11-23T02:42:53.8431544Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8432836Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73828
2022-11-23T02:42:53.8434004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73829
2022-11-23T02:42:53.8435220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8436125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8437272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8438214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8439401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8440268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8441414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8442364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8443264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8444242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8445564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8446982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8448014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8448926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8449597Z dist init r=1, world=2
2022-11-23T02:42:53.8450081Z dist init r=0, world=2
2022-11-23T02:42:53.8450522Z ok (3.913s)
2022-11-23T02:42:53.8451525Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8453036Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73907
2022-11-23T02:42:53.8454109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73908
2022-11-23T02:42:53.8455339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8456227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8457399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8458751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8459916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8460823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8461988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8462909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8463799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8464785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8466122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8467485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8468698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8469617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8470313Z dist init r=1, world=2
2022-11-23T02:42:53.8470811Z dist init r=0, world=2
2022-11-23T02:42:53.8471262Z ok (3.912s)
2022-11-23T02:42:53.8472269Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8473653Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73986
2022-11-23T02:42:53.8474713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73987
2022-11-23T02:42:53.8475991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8476868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8478004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8478949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8480110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8480980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8482140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8483077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8483980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8485036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8486545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8488011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8489091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8490050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8490781Z dist init r=0, world=2
2022-11-23T02:42:53.8491273Z dist init r=1, world=2
2022-11-23T02:42:53.8491712Z ok (4.012s)
2022-11-23T02:42:53.8492706Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8494160Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74065
2022-11-23T02:42:53.8495225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74066
2022-11-23T02:42:53.8496451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8497351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8498999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8499923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8501104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8502173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8503362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8504297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8505200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8506184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8507509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8508892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8509948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8510917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8511589Z dist init r=1, world=2
2022-11-23T02:42:53.8512079Z dist init r=0, world=2
2022-11-23T02:42:53.8512538Z ok (3.911s)
2022-11-23T02:42:53.8513509Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8514896Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74144
2022-11-23T02:42:53.8515952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74145
2022-11-23T02:42:53.8517212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8518083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8519295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8520255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8521600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8522507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8523686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8524620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8525522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8526504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8527853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8529219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8530294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8531246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8531945Z dist init r=0, world=2
2022-11-23T02:42:53.8532413Z dist init r=1, world=2
2022-11-23T02:42:53.8532852Z ok (4.012s)
2022-11-23T02:42:53.8533838Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8565053Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74223
2022-11-23T02:42:53.8566500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74224
2022-11-23T02:42:53.8567833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8568773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8569945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8570902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8572100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8573002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8574166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8575147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8576055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8577080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8578726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8580166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8581235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8582162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8582890Z dist init r=1, world=2
2022-11-23T02:42:53.8583389Z dist init r=0, world=2
2022-11-23T02:42:53.8583862Z ok (3.911s)
2022-11-23T02:42:53.8584940Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8586507Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74302
2022-11-23T02:42:53.8587613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74303
2022-11-23T02:42:53.8588856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8589769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8590945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8591885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8593064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8593984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8595170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8596130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8597012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8597989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8599347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8600748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8601949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8602899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8603623Z dist init r=0, world=2
2022-11-23T02:42:53.8604099Z dist init r=1, world=2
2022-11-23T02:42:53.8604564Z ok (4.112s)
2022-11-23T02:42:53.8605603Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8606963Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74381
2022-11-23T02:42:53.8608049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74382
2022-11-23T02:42:53.8609344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8610263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8611439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8612406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8613607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8614490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8615655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8616607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8617528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8618843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8620389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8621830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8622834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8623777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8624497Z dist init r=0, world=2
2022-11-23T02:42:53.8624993Z dist init r=1, world=2
2022-11-23T02:42:53.8625436Z ok (4.012s)
2022-11-23T02:42:53.8626468Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8627895Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74460
2022-11-23T02:42:53.8628977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74461
2022-11-23T02:42:53.8630231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8631159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8632297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8633242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8634422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8635486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8636669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8637609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8638528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8639536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8640883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8642246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8643321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8644299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8645029Z dist init r=0, world=2
2022-11-23T02:42:53.8645498Z dist init r=1, world=2
2022-11-23T02:42:53.8645964Z ok (3.912s)
2022-11-23T02:42:53.8647006Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8648422Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74539
2022-11-23T02:42:53.8649505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74540
2022-11-23T02:42:53.8650772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8651693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8652858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8653818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8655115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8656037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8657197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8658367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8659263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8660207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8661516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8662935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8664013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8664958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8665663Z dist init r=1, world=2
2022-11-23T02:42:53.8666162Z dist init r=0, world=2
2022-11-23T02:42:53.8666596Z ok (4.012s)
2022-11-23T02:42:53.8667641Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8669042Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74618
2022-11-23T02:42:53.8670283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74619
2022-11-23T02:42:53.8671511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8672427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8673614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8674575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8675752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8676667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8677856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8678774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8679689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8680691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8682050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8683437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8684549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8685538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8686277Z dist init r=1, world=2
2022-11-23T02:42:53.8686763Z dist init r=0, world=2
2022-11-23T02:42:53.8687244Z ok (4.112s)
2022-11-23T02:42:53.8688284Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8689823Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74697
2022-11-23T02:42:53.8690935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74698
2022-11-23T02:42:53.8692214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8693140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8694312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8695290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8696489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8697389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8698920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8699874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8700789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8701771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8703129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8704554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8705803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8706739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8707463Z dist init r=1, world=2
2022-11-23T02:42:53.8707962Z dist init r=0, world=2
2022-11-23T02:42:53.8708392Z ok (4.012s)
2022-11-23T02:42:53.8709431Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8710839Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74776
2022-11-23T02:42:53.8711923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74777
2022-11-23T02:42:53.8713176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8714089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8715243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8716136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8717325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8718258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8719491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8720399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8721315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8722326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8723817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8725240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8726303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8727269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8727963Z dist init r=1, world=2
2022-11-23T02:42:53.8728463Z dist init r=0, world=2
2022-11-23T02:42:53.8728921Z ok (4.012s)
2022-11-23T02:42:53.8729928Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8731346Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74855
2022-11-23T02:42:53.8732428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74856
2022-11-23T02:42:53.8733688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8734581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8735769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8736717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8738138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8739183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8740374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8741329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8742230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8743204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8744552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8745950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8747011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8747984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8748702Z dist init r=1, world=2
2022-11-23T02:42:53.8749185Z dist init r=0, world=2
2022-11-23T02:42:53.8749634Z ok (4.011s)
2022-11-23T02:42:53.8750676Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8752113Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74934
2022-11-23T02:42:53.8753186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74935
2022-11-23T02:42:53.8754432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8755332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8756521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8757440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8758786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8759703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8760888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8761819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8762721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8763735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8765067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8767426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8768513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8769462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8770163Z dist init r=0, world=2
2022-11-23T02:42:53.8770662Z dist init r=1, world=2
2022-11-23T02:42:53.8771136Z ok (4.013s)
2022-11-23T02:42:53.8772153Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8773568Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75013
2022-11-23T02:42:53.8774755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75014
2022-11-23T02:42:53.8776024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8776878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8778465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8779404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8780563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8781410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8782536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8783478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8784470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8785463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8786811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8788227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8789273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8790259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8790984Z dist init r=0, world=2
2022-11-23T02:42:53.8791479Z dist init r=1, world=2
2022-11-23T02:42:53.8791932Z ok (4.013s)
2022-11-23T02:42:53.8792959Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8794535Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75092
2022-11-23T02:42:53.8795609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75093
2022-11-23T02:42:53.8796883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8797809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8798993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8799922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8801141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8802028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8803217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8804152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8805071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8806065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8807407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8808846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8810103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8811077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8811777Z dist init r=0, world=2
2022-11-23T02:42:53.8812279Z dist init r=1, world=2
2022-11-23T02:42:53.8812739Z ok (3.912s)
2022-11-23T02:42:53.8813740Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8815176Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75171
2022-11-23T02:42:53.8816259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75172
2022-11-23T02:42:53.8817525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8818658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8819865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8820827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8822033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8822911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8824076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8825030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8825916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8826951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8828454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8829901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8830940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8831915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8832644Z dist init r=1, world=2
2022-11-23T02:42:53.8833140Z dist init r=0, world=2
2022-11-23T02:42:53.8833593Z ok (3.912s)
2022-11-23T02:42:53.8834635Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8836049Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75250
2022-11-23T02:42:53.8837108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75251
2022-11-23T02:42:53.8838374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8839271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8840461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8841407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8842594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8843704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8844998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8845957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8846850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8847824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8849090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8850456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8851506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8852473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8853156Z dist init r=0, world=2
2022-11-23T02:42:53.8853632Z dist init r=1, world=2
2022-11-23T02:42:53.8854086Z ok (3.912s)
2022-11-23T02:42:53.8855087Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8856493Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75329
2022-11-23T02:42:53.8857840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75330
2022-11-23T02:42:53.8859640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8860523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8861702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8862638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8863964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8864844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8866004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8866947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8867829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8868837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8870183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8871586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8872631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8873597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8874308Z dist init r=0, world=2
2022-11-23T02:42:53.8874768Z dist init r=1, world=2
2022-11-23T02:42:53.8875220Z ok (4.012s)
2022-11-23T02:42:53.8876252Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8877636Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75408
2022-11-23T02:42:53.8878824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75409
2022-11-23T02:42:53.8880102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8881009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8882198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8883136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8884406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8885339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8886494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8887462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8888377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8889403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8890713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8892126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8893191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8894153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8894861Z dist init r=0, world=2
2022-11-23T02:42:53.8895351Z dist init r=1, world=2
2022-11-23T02:42:53.8895815Z ok (3.912s)
2022-11-23T02:42:53.8896806Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8898728Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75487
2022-11-23T02:42:53.8899847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75488
2022-11-23T02:42:53.8901113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8902028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8903202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8904142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8905321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8906227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8907414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8908372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8909260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8910270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8911638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8913061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8914261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8915212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8915941Z dist init r=0, world=2
2022-11-23T02:42:53.8916408Z dist init r=1, world=2
2022-11-23T02:42:53.8916871Z ok (3.912s)
2022-11-23T02:42:53.8917885Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8919286Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75566
2022-11-23T02:42:53.8920346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75567
2022-11-23T02:42:53.8921594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8922532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8923701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8924659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8925861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8926768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8927915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8928867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8929785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8930805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8932288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8933740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8934819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8935756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8936481Z dist init r=0, world=2
2022-11-23T02:42:53.8936967Z dist init r=1, world=2
2022-11-23T02:42:53.8937407Z ok (4.012s)
2022-11-23T02:42:53.8938734Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8940142Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75645
2022-11-23T02:42:53.8941241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75646
2022-11-23T02:42:53.8942491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8943407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8944581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8945531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8946697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8947779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8948974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8949926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8950817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8951864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8953246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8954624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8955695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8956648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8957353Z dist init r=0, world=2
2022-11-23T02:42:53.8957804Z dist init r=1, world=2
2022-11-23T02:42:53.8958245Z ok (4.012s)
2022-11-23T02:42:53.8959242Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8960593Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75724
2022-11-23T02:42:53.8961669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75725
2022-11-23T02:42:53.8962917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8963815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8964968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8965907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8967244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8968165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8969312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8970255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8971161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8972108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8973477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8974868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8975919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8976827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8977513Z dist init r=0, world=2
2022-11-23T02:42:53.8978365Z dist init r=1, world=2
2022-11-23T02:42:53.8978816Z ok (4.012s)
2022-11-23T02:42:53.8979804Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.8981174Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75803
2022-11-23T02:42:53.8982411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75804
2022-11-23T02:42:53.8983640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8984613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8985791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8986742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8987908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.8988808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.8989988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.8990900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.8991810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.8992801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.8994126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8995442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.8996504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.8997443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.8998151Z dist init r=1, world=2
2022-11-23T02:42:53.8998606Z dist init r=0, world=2
2022-11-23T02:42:53.8999070Z ok (4.012s)
2022-11-23T02:42:53.9000057Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9001590Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75882
2022-11-23T02:42:53.9002664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75883
2022-11-23T02:42:53.9003931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9004827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9005973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9006934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9008144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9009016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9010202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9011139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9012045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9013037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9014357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9015794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9017000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9018314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9019027Z dist init r=1, world=2
2022-11-23T02:42:53.9019509Z dist init r=0, world=2
2022-11-23T02:42:53.9019945Z ok (4.012s)
2022-11-23T02:42:53.9020944Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9022353Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75961
2022-11-23T02:42:53.9023416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75962
2022-11-23T02:42:53.9024666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9025586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9026762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9027676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9028857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9029763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9030941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9031823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9032726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9033728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9035089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9036626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9037689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9038633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9039323Z dist init r=1, world=2
2022-11-23T02:42:53.9039804Z dist init r=0, world=2
2022-11-23T02:42:53.9040262Z ok (4.012s)
2022-11-23T02:42:53.9041244Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9042608Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76040
2022-11-23T02:42:53.9043664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76041
2022-11-23T02:42:53.9044922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9045838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9046961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9047886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9049070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9049938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9051278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9052190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9053088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9054055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9055396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9056781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9057862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9059063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9059770Z dist init r=1, world=2
2022-11-23T02:42:53.9060256Z dist init r=0, world=2
2022-11-23T02:42:53.9060699Z ok (4.012s)
2022-11-23T02:42:53.9061701Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9063060Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76119
2022-11-23T02:42:53.9064145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76120
2022-11-23T02:42:53.9065404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9066309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9067473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9068386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9069715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9070626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9071796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9072754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9073645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9074630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9075931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9077347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9078406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9079363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9080064Z dist init r=0, world=2
2022-11-23T02:42:53.9080551Z dist init r=1, world=2
2022-11-23T02:42:53.9081001Z ok (4.012s)
2022-11-23T02:42:53.9081979Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9083339Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76198
2022-11-23T02:42:53.9084665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76199
2022-11-23T02:42:53.9085910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9086800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9087961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9088847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9090008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9090922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9092146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9093141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9094017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9094983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9096330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9097763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9099059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9100050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9100767Z dist init r=0, world=2
2022-11-23T02:42:53.9101252Z dist init r=1, world=2
2022-11-23T02:42:53.9101674Z ok (4.012s)
2022-11-23T02:42:53.9102656Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9104211Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76277
2022-11-23T02:42:53.9105252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76278
2022-11-23T02:42:53.9106486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9107386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9108552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9109436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9110577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9111468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9112596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9113549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9114449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9115457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9116735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9118142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9119358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9120301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9120987Z dist init r=1, world=2
2022-11-23T02:42:53.9121475Z dist init r=0, world=2
2022-11-23T02:42:53.9121938Z ok (4.012s)
2022-11-23T02:42:53.9122926Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9124321Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76356
2022-11-23T02:42:53.9125351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76357
2022-11-23T02:42:53.9126620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9127514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9128704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9129651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9130822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9131706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9132878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9133802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9134663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9135661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9137005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9138837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9139885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9140852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9141550Z dist init r=1, world=2
2022-11-23T02:42:53.9142020Z dist init r=0, world=2
2022-11-23T02:42:53.9142480Z ok (4.112s)
2022-11-23T02:42:53.9143481Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9144871Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76435
2022-11-23T02:42:53.9146024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76436
2022-11-23T02:42:53.9147284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9148184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9149361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9150261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9151442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9152346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9153699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9154642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9155541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9156537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9157849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9159283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9160325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9161264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9161974Z dist init r=1, world=2
2022-11-23T02:42:53.9162459Z dist init r=0, world=2
2022-11-23T02:42:53.9162923Z ok (4.012s)
2022-11-23T02:42:53.9163882Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9165248Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76514
2022-11-23T02:42:53.9166312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76515
2022-11-23T02:42:53.9167589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9168477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9169653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9170603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9171761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9172826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9174013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9174953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9175826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9176814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9178422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9179861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9180903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9181843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9182555Z dist init r=1, world=2
2022-11-23T02:42:53.9183022Z dist init r=0, world=2
2022-11-23T02:42:53.9183473Z ok (4.012s)
2022-11-23T02:42:53.9184458Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9185844Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76593
2022-11-23T02:42:53.9186893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76594
2022-11-23T02:42:53.9188319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9189229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9190379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9191320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9192504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9193400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9194551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9195492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9196390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9197369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9198720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9200110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9201157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9202072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9202801Z dist init r=0, world=2
2022-11-23T02:42:53.9203281Z dist init r=1, world=2
2022-11-23T02:42:53.9203710Z ok (4.012s)
2022-11-23T02:42:53.9204666Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9206200Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76672
2022-11-23T02:42:53.9207294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76673
2022-11-23T02:42:53.9208530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9209418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9210601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9211461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9212621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9213531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9214697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9215629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9216498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9217519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9219265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9220642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9221690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9222830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9223571Z dist init r=1, world=2
2022-11-23T02:42:53.9224042Z dist init r=0, world=2
2022-11-23T02:42:53.9224519Z ok (4.012s)
2022-11-23T02:42:53.9225516Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9226849Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76751
2022-11-23T02:42:53.9227986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76752
2022-11-23T02:42:53.9229265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9230145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9231292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9232233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9233407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9234311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9235432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9236374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9237259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9238197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9239571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9241142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9242212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9243139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9243833Z dist init r=1, world=2
2022-11-23T02:42:53.9244302Z dist init r=0, world=2
2022-11-23T02:42:53.9244748Z ok (4.112s)
2022-11-23T02:42:53.9245689Z   test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76830
2022-11-23T02:42:53.9246825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76831
2022-11-23T02:42:53.9248061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9248941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9250119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9251052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9252184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9253077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9254236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9255171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9256163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9257150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9258779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9260203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9261226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9262192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9264843Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9266548Z   warnings.warn(
2022-11-23T02:42:53.9268891Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9270478Z   warnings.warn(
2022-11-23T02:42:53.9270942Z dist init r=1, world=2
2022-11-23T02:42:53.9271423Z dist init r=0, world=2
2022-11-23T02:42:53.9271884Z ok (3.812s)
2022-11-23T02:42:53.9272770Z   test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76909
2022-11-23T02:42:53.9273925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76910
2022-11-23T02:42:53.9275358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9276240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9277397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9278338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9279546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9280406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9281578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9282490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9283407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9284360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9285773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9287178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9288232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9289152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9291920Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9293529Z   warnings.warn(
2022-11-23T02:42:53.9295905Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9297488Z   warnings.warn(
2022-11-23T02:42:53.9298226Z dist init r=1, world=2
2022-11-23T02:42:53.9298724Z dist init r=0, world=2
2022-11-23T02:42:53.9299170Z ok (3.911s)
2022-11-23T02:42:53.9300059Z   test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76988
2022-11-23T02:42:53.9301184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76989
2022-11-23T02:42:53.9302438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9303313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9304496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9305461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9306666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9307546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9308899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9309882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9310796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9311752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9313111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9313925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9314404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9314857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9316952Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9317177Z   warnings.warn(
2022-11-23T02:42:53.9319241Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9319665Z   warnings.warn(
2022-11-23T02:42:53.9319859Z dist init r=1, world=2
2022-11-23T02:42:53.9320066Z dist init r=0, world=2
2022-11-23T02:42:53.9320247Z ok (3.911s)
2022-11-23T02:42:53.9320938Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9321882Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77067
2022-11-23T02:42:53.9322318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77068
2022-11-23T02:42:53.9323070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9323427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9324170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9324554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9325323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9325660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9326419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9326788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9327262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9327755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9328528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9329460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9329944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9330404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9331707Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9331908Z   warnings.warn(
2022-11-23T02:42:53.9333147Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9333375Z   warnings.warn(
2022-11-23T02:42:53.9333585Z dist init r=0, world=2
2022-11-23T02:42:53.9333763Z dist init r=1, world=2
2022-11-23T02:42:53.9333944Z ok (4.011s)
2022-11-23T02:42:53.9334599Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9335525Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77146
2022-11-23T02:42:53.9335969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77147
2022-11-23T02:42:53.9336724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9337188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9338298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9338696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9339420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9339771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9340552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9340930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9341418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9341913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9342728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9343550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9344016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9344438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9345700Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9345922Z   warnings.warn(
2022-11-23T02:42:53.9347152Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9347377Z   warnings.warn(
2022-11-23T02:42:53.9347579Z dist init r=1, world=2
2022-11-23T02:42:53.9347941Z dist init r=0, world=2
2022-11-23T02:42:53.9348139Z ok (4.011s)
2022-11-23T02:42:53.9348753Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9349686Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77225
2022-11-23T02:42:53.9350139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77226
2022-11-23T02:42:53.9350892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9351244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9352000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9352385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9353137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9353485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9354220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9354600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9355085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9355565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9356540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9357352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9357817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9358273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9359569Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9359754Z   warnings.warn(
2022-11-23T02:42:53.9361037Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9361254Z   warnings.warn(
2022-11-23T02:42:53.9361461Z dist init r=0, world=2
2022-11-23T02:42:53.9361666Z dist init r=1, world=2
2022-11-23T02:42:53.9361849Z ok (4.011s)
2022-11-23T02:42:53.9362476Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9363393Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77304
2022-11-23T02:42:53.9363823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77305
2022-11-23T02:42:53.9364567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9364912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9365684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9366056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9366922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9367289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9368077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9368453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9368908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9369389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9370211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9371048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9371511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9371970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9373244Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9373460Z   warnings.warn(
2022-11-23T02:42:53.9374717Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9375047Z   warnings.warn(
2022-11-23T02:42:53.9375232Z dist init r=1, world=2
2022-11-23T02:42:53.9375437Z dist init r=0, world=2
2022-11-23T02:42:53.9375623Z ok (3.913s)
2022-11-23T02:42:53.9376271Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9377222Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77383
2022-11-23T02:42:53.9377650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77384
2022-11-23T02:42:53.9378739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9379085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9379844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9380227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9380970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9381315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9382074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9382439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9382908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9383384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9384209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9385057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9385684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9386155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9387435Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9387645Z   warnings.warn(
2022-11-23T02:42:53.9388916Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9389131Z   warnings.warn(
2022-11-23T02:42:53.9389331Z dist init r=0, world=2
2022-11-23T02:42:53.9389512Z dist init r=1, world=2
2022-11-23T02:42:53.9389697Z ok (3.912s)
2022-11-23T02:42:53.9390357Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9391301Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77462
2022-11-23T02:42:53.9391731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77463
2022-11-23T02:42:53.9392486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9392833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9393717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9394087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9394819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9395161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9395933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9396304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9396781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9397252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9398070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9398868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9399341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9399775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9401062Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9401275Z   warnings.warn(
2022-11-23T02:42:53.9402537Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9402753Z   warnings.warn(
2022-11-23T02:42:53.9402962Z dist init r=1, world=2
2022-11-23T02:42:53.9403174Z dist init r=0, world=2
2022-11-23T02:42:53.9403358Z ok (4.012s)
2022-11-23T02:42:53.9404131Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9405040Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77541
2022-11-23T02:42:53.9405475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77542
2022-11-23T02:42:53.9406238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9406578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9407362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9407730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9408484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9408823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9409550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9409928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9410392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9410865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9411687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9412621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9413087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9413551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9414801Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9415019Z   warnings.warn(
2022-11-23T02:42:53.9416254Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9416488Z   warnings.warn(
2022-11-23T02:42:53.9416696Z dist init r=0, world=2
2022-11-23T02:42:53.9416908Z dist init r=1, world=2
2022-11-23T02:42:53.9417082Z ok (4.011s)
2022-11-23T02:42:53.9417748Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9419240Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77620
2022-11-23T02:42:53.9419488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77621
2022-11-23T02:42:53.9419887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9420066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9420453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9420654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9421221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9421413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9421793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9421986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9422213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9422458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9422860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9423260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9423497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9423724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9424344Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9424461Z   warnings.warn(
2022-11-23T02:42:53.9425079Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9425280Z   warnings.warn(
2022-11-23T02:42:53.9425376Z dist init r=1, world=2
2022-11-23T02:42:53.9425484Z dist init r=0, world=2
2022-11-23T02:42:53.9425586Z ok (3.911s)
2022-11-23T02:42:53.9425920Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9426368Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77699
2022-11-23T02:42:53.9426586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77700
2022-11-23T02:42:53.9426955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9427130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9427490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9427686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9428055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9428228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9428600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9428787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9429031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9429272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9429672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9430049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9430363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9430597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9431222Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9431337Z   warnings.warn(
2022-11-23T02:42:53.9431953Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9432067Z   warnings.warn(
2022-11-23T02:42:53.9432179Z dist init r=1, world=2
2022-11-23T02:42:53.9432290Z dist init r=0, world=2
2022-11-23T02:42:53.9432374Z ok (3.911s)
2022-11-23T02:42:53.9432706Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9433154Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77778
2022-11-23T02:42:53.9433373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77779
2022-11-23T02:42:53.9433741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9433917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9434295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9434541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9434887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9435066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9435440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9435627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9435869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9436110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9436507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9436904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9437133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9437343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9437962Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9438077Z   warnings.warn(
2022-11-23T02:42:53.9438692Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9438808Z   warnings.warn(
2022-11-23T02:42:53.9438923Z dist init r=0, world=2
2022-11-23T02:42:53.9439033Z dist init r=1, world=2
2022-11-23T02:42:53.9439135Z ok (3.911s)
2022-11-23T02:42:53.9439495Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9439952Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77857
2022-11-23T02:42:53.9440173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77858
2022-11-23T02:42:53.9440544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9440718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9441098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9441290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9441653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9441831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9442185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9442371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9442611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9442853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9443255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9443709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9443937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9444163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9444276Z dist init r=0, world=2
2022-11-23T02:42:53.9444367Z dist init r=1, world=2
2022-11-23T02:42:53.9444469Z ok (4.011s)
2022-11-23T02:42:53.9444802Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9445247Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77936
2022-11-23T02:42:53.9445465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77937
2022-11-23T02:42:53.9445841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9446017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9446396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9446566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9446934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9447106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9447477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9447664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9447907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9448153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9448594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9448996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9449205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9449432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9449544Z dist init r=1, world=2
2022-11-23T02:42:53.9449653Z dist init r=0, world=2
2022-11-23T02:42:53.9449754Z ok (3.911s)
2022-11-23T02:42:53.9450074Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9450528Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78015
2022-11-23T02:42:53.9450750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78016
2022-11-23T02:42:53.9451101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9451276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9451651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9451840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9452204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9452425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9452802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9452987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9453232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9453455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9453859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9454257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9454487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9454717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9454835Z dist init r=1, world=2
2022-11-23T02:42:53.9454943Z dist init r=0, world=2
2022-11-23T02:42:53.9455046Z ok (3.911s)
2022-11-23T02:42:53.9455346Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9455790Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78094
2022-11-23T02:42:53.9456004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78095
2022-11-23T02:42:53.9456374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9456549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9456928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9457123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9457534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9457713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9458398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9458597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9458837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9459080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9459487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9459882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9460113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9460338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9460451Z dist init r=1, world=2
2022-11-23T02:42:53.9460544Z dist init r=0, world=2
2022-11-23T02:42:53.9460645Z ok (3.911s)
2022-11-23T02:42:53.9460966Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9461410Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78173
2022-11-23T02:42:53.9461628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78174
2022-11-23T02:42:53.9462093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9462267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9462646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9462816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9463179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9463351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9463717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9463904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9464150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9464388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9464789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9465182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9465391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9465615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9465727Z dist init r=1, world=2
2022-11-23T02:42:53.9465834Z dist init r=0, world=2
2022-11-23T02:42:53.9465935Z ok (4.011s)
2022-11-23T02:42:53.9466250Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9466700Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78252
2022-11-23T02:42:53.9466977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78253
2022-11-23T02:42:53.9467354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9467510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9467887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9468076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9468444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9468621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9468993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9469181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9469423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9469644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9470038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9470430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9470658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9470936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9471048Z dist init r=0, world=2
2022-11-23T02:42:53.9471155Z dist init r=1, world=2
2022-11-23T02:42:53.9471256Z ok (4.011s)
2022-11-23T02:42:53.9471578Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9472021Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78331
2022-11-23T02:42:53.9472236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78332
2022-11-23T02:42:53.9472604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9472779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9473158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9473346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9473713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9473885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9474237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9474423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9474661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9474897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9475293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9475688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9475962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9476193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9476304Z dist init r=0, world=2
2022-11-23T02:42:53.9476395Z dist init r=1, world=2
2022-11-23T02:42:53.9476491Z ok (3.911s)
2022-11-23T02:42:53.9476824Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9477269Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78410
2022-11-23T02:42:53.9477488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78411
2022-11-23T02:42:53.9477854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9478031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9478408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9478578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9478946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9479116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9479484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9479739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9479980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9480216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9480616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9481007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9481216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9481440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9481551Z dist init r=0, world=2
2022-11-23T02:42:53.9481660Z dist init r=1, world=2
2022-11-23T02:42:53.9481760Z ok (4.011s)
2022-11-23T02:42:53.9482085Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9482536Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78489
2022-11-23T02:42:53.9482754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78490
2022-11-23T02:42:53.9483104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9483277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9483656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9483844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9484250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9484426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9484846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9485040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9485281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9485503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9485902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9486294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9486524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9486751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9486863Z dist init r=1, world=2
2022-11-23T02:42:53.9486971Z dist init r=0, world=2
2022-11-23T02:42:53.9487074Z ok (4.011s)
2022-11-23T02:42:53.9487373Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9487818Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78568
2022-11-23T02:42:53.9488034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78569
2022-11-23T02:42:53.9488405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9488577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9489007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9489197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9489561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9489731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9490083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9490268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9490509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9490749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9491149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9491542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9491768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9491993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9492087Z dist init r=0, world=2
2022-11-23T02:42:53.9492194Z dist init r=1, world=2
2022-11-23T02:42:53.9492294Z ok (4.013s)
2022-11-23T02:42:53.9492617Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9492935Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78647
2022-11-23T02:42:53.9493154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78648
2022-11-23T02:42:53.9493528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9493752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9494135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9494305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9494669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9494838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9495206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9495396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9495639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9495881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9496275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9496666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9496875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9497098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9497210Z dist init r=1, world=2
2022-11-23T02:42:53.9497317Z dist init r=0, world=2
2022-11-23T02:42:53.9497472Z ok (4.413s)
2022-11-23T02:42:53.9497794Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9498284Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78730
2022-11-23T02:42:53.9498506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78731
2022-11-23T02:42:53.9498865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9499037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9499413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9499600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9499967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9500139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9500511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9500696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9500917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9501156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9501554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9501946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9502177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9502402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9502516Z dist init r=1, world=2
2022-11-23T02:42:53.9502696Z dist init r=0, world=2
2022-11-23T02:42:53.9502787Z ok (3.912s)
2022-11-23T02:42:53.9503109Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9503422Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78809
2022-11-23T02:42:53.9503636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78810
2022-11-23T02:42:53.9504012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9504188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9504566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9504758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9505122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9505275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9505643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9505829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9506070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9506309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9506775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9507170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9507396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9507618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9507713Z dist init r=1, world=2
2022-11-23T02:42:53.9507821Z dist init r=0, world=2
2022-11-23T02:42:53.9507921Z ok (4.412s)
2022-11-23T02:42:53.9508242Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9508558Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78892
2022-11-23T02:42:53.9508778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78893
2022-11-23T02:42:53.9509155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9509330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9509691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9509880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9510243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9510414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9510782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9510971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9511257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9511503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9511901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9512279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9512507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9512729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9512845Z dist init r=0, world=2
2022-11-23T02:42:53.9512954Z dist init r=1, world=2
2022-11-23T02:42:53.9513055Z ok (3.913s)
2022-11-23T02:42:53.9513380Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9513699Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78971
2022-11-23T02:42:53.9513895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78972
2022-11-23T02:42:53.9514267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9514440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9514814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9515052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9515415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9515587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9515965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9516152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9516374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9516613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9517010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9517405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9517635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9517864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9517977Z dist init r=1, world=2
2022-11-23T02:42:53.9518086Z dist init r=0, world=2
2022-11-23T02:42:53.9518168Z ok (4.513s)
2022-11-23T02:42:53.9518494Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9518808Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79054
2022-11-23T02:42:53.9519024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79055
2022-11-23T02:42:53.9519395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9519570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9519996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9520189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9520555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9520708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9521079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9521261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9521500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9521743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9522142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9522535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9522762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9522986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9523079Z dist init r=0, world=2
2022-11-23T02:42:53.9523186Z dist init r=1, world=2
2022-11-23T02:42:53.9523286Z ok (4.012s)
2022-11-23T02:42:53.9523612Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9523978Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79133
2022-11-23T02:42:53.9524195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79134
2022-11-23T02:42:53.9524570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9524742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9525100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9525286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9525652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9525824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9526198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9526384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9526627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9526867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9527259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9527633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9527859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9528084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9528199Z dist init r=1, world=2
2022-11-23T02:42:53.9528306Z dist init r=0, world=2
2022-11-23T02:42:53.9528406Z ok (4.413s)
2022-11-23T02:42:53.9528777Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9529097Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79216
2022-11-23T02:42:53.9529294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79217
2022-11-23T02:42:53.9529668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9529840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9530214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9530406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9530773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9530948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9531320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9531505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9531727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9531966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9532362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9532811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9533043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9533271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9533382Z dist init r=0, world=2
2022-11-23T02:42:53.9533490Z dist init r=1, world=2
2022-11-23T02:42:53.9533573Z ok (3.912s)
2022-11-23T02:42:53.9533887Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9534201Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79295
2022-11-23T02:42:53.9534418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79296
2022-11-23T02:42:53.9534792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9534965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9535348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9535536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9535899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9536052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9536422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9536605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9536850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9537090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9537532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9538096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9538330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9538539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9538651Z dist init r=1, world=2
2022-11-23T02:42:53.9538760Z dist init r=0, world=2
2022-11-23T02:42:53.9538861Z ok (4.413s)
2022-11-23T02:42:53.9539177Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9539497Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79378
2022-11-23T02:42:53.9539716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79379
2022-11-23T02:42:53.9540095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9540267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9540625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9540811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9541174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9541464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9541836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9542023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9542264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9542504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9542883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9543275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9543503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9543729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9543841Z dist init r=1, world=2
2022-11-23T02:42:53.9543949Z dist init r=0, world=2
2022-11-23T02:42:53.9544048Z ok (4.513s)
2022-11-23T02:42:53.9544363Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9544658Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79461
2022-11-23T02:42:53.9544873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79462
2022-11-23T02:42:53.9545243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9545415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9545790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9545980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9546403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9546584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9546957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9547123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9547361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9547597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9547990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9548384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9548614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9548838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9548949Z dist init r=1, world=2
2022-11-23T02:42:53.9549057Z dist init r=0, world=2
2022-11-23T02:42:53.9549139Z ok (4.413s)
2022-11-23T02:42:53.9549451Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9549765Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79544
2022-11-23T02:42:53.9550035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79545
2022-11-23T02:42:53.9550405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9550582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9550957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9551147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9551497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9551670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9552041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9552226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9552472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9552711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9553109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9553500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9553725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9553932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9554044Z dist init r=0, world=2
2022-11-23T02:42:53.9554153Z dist init r=1, world=2
2022-11-23T02:42:53.9554253Z ok (4.513s)
2022-11-23T02:42:53.9554565Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9555045Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79627
2022-11-23T02:42:53.9555267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79628
2022-11-23T02:42:53.9555633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9555788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9556165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9556352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9556711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9556886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9557254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9557439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9557680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9557917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9558296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9558676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9558899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9559203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9559829Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9559945Z   warnings.warn(
2022-11-23T02:42:53.9560561Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9560671Z   warnings.warn(
2022-11-23T02:42:53.9560781Z dist init r=1, world=2
2022-11-23T02:42:53.9560872Z dist init r=0, world=2
2022-11-23T02:42:53.9560975Z ok (4.513s)
2022-11-23T02:42:53.9561282Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9561715Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79710
2022-11-23T02:42:53.9561934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79711
2022-11-23T02:42:53.9562302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9562476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9562850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9563020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9563381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9563558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9563927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9564165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9564414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9564653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9565051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9565444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9565652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9565881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9566895Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9567013Z   warnings.warn(
2022-11-23T02:42:53.9568004Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9568163Z   warnings.warn(
2022-11-23T02:42:53.9568781Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9568893Z   warnings.warn(
2022-11-23T02:42:53.9569504Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9569613Z   warnings.warn(
2022-11-23T02:42:53.9569724Z dist init r=1, world=2
2022-11-23T02:42:53.9569816Z dist init r=0, world=2
2022-11-23T02:42:53.9569918Z ok (4.513s)
2022-11-23T02:42:53.9570226Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9570657Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79793
2022-11-23T02:42:53.9570877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79794
2022-11-23T02:42:53.9571247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9571421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9571799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9571988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9572338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9572513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9572885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9573120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9573366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9573609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9574003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9574396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9574620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9574830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9574942Z dist init r=1, world=2
2022-11-23T02:42:53.9575050Z dist init r=0, world=2
2022-11-23T02:42:53.9575150Z ok (3.811s)
2022-11-23T02:42:53.9575460Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9575890Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79872
2022-11-23T02:42:53.9576110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79873
2022-11-23T02:42:53.9576474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9576629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9577004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9577245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9577608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9577782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9578321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9578507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9578749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9578971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9579371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9579767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9579998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9580220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9580331Z dist init r=0, world=2
2022-11-23T02:42:53.9580442Z dist init r=1, world=2
2022-11-23T02:42:53.9580542Z ok (3.912s)
2022-11-23T02:42:53.9580827Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9581252Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79951
2022-11-23T02:42:53.9581468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79952
2022-11-23T02:42:53.9581843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9582018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9582464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9582662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9583026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9583201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9583551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9583737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9583980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9584259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9584660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9585050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9585278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9585501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9585613Z dist init r=0, world=2
2022-11-23T02:42:53.9585704Z dist init r=1, world=2
2022-11-23T02:42:53.9585805Z ok (4.413s)
2022-11-23T02:42:53.9586102Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9586603Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80034
2022-11-23T02:42:53.9586824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80035
2022-11-23T02:42:53.9587192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9587365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9587741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9587913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9588273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9588446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9588816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9589004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9589244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9589485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9589880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9590275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9590483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9590710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9591771Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9591890Z   warnings.warn(
2022-11-23T02:42:53.9592887Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9593002Z   warnings.warn(
2022-11-23T02:42:53.9593113Z dist init r=0, world=2
2022-11-23T02:42:53.9593220Z dist init r=1, world=2
2022-11-23T02:42:53.9593321Z ok (4.513s)
2022-11-23T02:42:53.9593621Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9594030Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80117
2022-11-23T02:42:53.9594248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80118
2022-11-23T02:42:53.9594612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9594785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9595216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9595403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9595770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9595940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9596311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9596476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9596718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9596959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9597354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9597752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9597981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9598207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9598319Z dist init r=0, world=2
2022-11-23T02:42:53.9598410Z dist init r=1, world=2
2022-11-23T02:42:53.9598512Z ok (4.412s)
2022-11-23T02:42:53.9598807Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9599234Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80200
2022-11-23T02:42:53.9599454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80201
2022-11-23T02:42:53.9599819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9600050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9600433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9600619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9600959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9601132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9601498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9601682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9601926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9602166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9602562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9602946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9603171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9603377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9604390Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9604587Z   warnings.warn(
2022-11-23T02:42:53.9605584Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9605696Z   warnings.warn(
2022-11-23T02:42:53.9605808Z dist init r=1, world=2
2022-11-23T02:42:53.9605917Z dist init r=0, world=2
2022-11-23T02:42:53.9606024Z ok (4.413s)
2022-11-23T02:42:53.9606257Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9606568Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80283
2022-11-23T02:42:53.9606768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80284
2022-11-23T02:42:53.9607136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9607309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9607687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9607876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9608245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9608421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9608794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9609031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9609263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9609501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9609899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9610293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9610521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9610751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9610865Z dist init r=1, world=2
2022-11-23T02:42:53.9610973Z dist init r=0, world=2
2022-11-23T02:42:53.9611057Z ok (4.112s)
2022-11-23T02:42:53.9611291Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9611596Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80362
2022-11-23T02:42:53.9611810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80363
2022-11-23T02:42:53.9612185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9612358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9612733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9612971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9613321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9613494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9613862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9614047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9614289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9614527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9614920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9615319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9615548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9615755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9615866Z dist init r=1, world=2
2022-11-23T02:42:53.9615972Z dist init r=0, world=2
2022-11-23T02:42:53.9616072Z ok (4.212s)
2022-11-23T02:42:53.9616417Z   test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80441
2022-11-23T02:42:53.9616633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80442
2022-11-23T02:42:53.9617003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9617181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9617541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9617775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9618328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9618502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9618876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9619059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9619301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9619546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9619939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9620319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9620547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9620770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9621782Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9621976Z   warnings.warn(
2022-11-23T02:42:53.9622974Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9623085Z   warnings.warn(
2022-11-23T02:42:53.9623319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9623550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9623778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9624016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9624111Z dist init r=0, world=2
2022-11-23T02:42:53.9624221Z dist init r=1, world=2
2022-11-23T02:42:53.9624323Z ok (4.512s)
2022-11-23T02:42:53.9624674Z   test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80524
2022-11-23T02:42:53.9624889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80525
2022-11-23T02:42:53.9625265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9625439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9625817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9625992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9626353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9626584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9626972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9627158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9627400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9627639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9628037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9628429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9628643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9628874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9629881Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9629996Z   warnings.warn(
2022-11-23T02:42:53.9630992Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9631154Z   warnings.warn(
2022-11-23T02:42:53.9631390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9631621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9631850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9632082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9632193Z dist init r=1, world=2
2022-11-23T02:42:53.9632287Z dist init r=0, world=2
2022-11-23T02:42:53.9632387Z ok (4.512s)
2022-11-23T02:42:53.9632727Z   test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80607
2022-11-23T02:42:53.9632944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80608
2022-11-23T02:42:53.9633323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9633498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9633872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9634059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9634401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9634572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9634952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9635139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9635425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9635673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9636071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9636462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9636689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9636897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9637918Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9638033Z   warnings.warn(
2022-11-23T02:42:53.9639026Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9639184Z   warnings.warn(
2022-11-23T02:42:53.9639418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9639657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9639887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9640115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:42:53.9640225Z dist init r=0, world=2
2022-11-23T02:42:53.9640320Z dist init r=1, world=2
2022-11-23T02:42:53.9640420Z ok (4.411s)
2022-11-23T02:42:53.9640780Z   test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80690
2022-11-23T02:42:53.9640997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80691
2022-11-23T02:42:53.9641375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9641551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9641929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9642117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9642478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9642631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9643005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9643191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9643433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9643677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9644119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9644522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9644749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9644955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9645067Z dist init r=1, world=2
2022-11-23T02:42:53.9645175Z dist init r=0, world=2
2022-11-23T02:42:53.9645274Z ok (4.413s)
2022-11-23T02:42:53.9645642Z   test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80773
2022-11-23T02:42:53.9645863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80774
2022-11-23T02:42:53.9646240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9646413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9646773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9646961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9647323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9647494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9647866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9648103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9648347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9648590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9648986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9649360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9649585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9649807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9650430Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9650549Z   warnings.warn(
2022-11-23T02:42:53.9651168Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9651281Z   warnings.warn(
2022-11-23T02:42:53.9651393Z dist init r=1, world=2
2022-11-23T02:42:53.9651501Z dist init r=0, world=2
2022-11-23T02:42:53.9651583Z ok (4.413s)
2022-11-23T02:42:53.9651936Z   test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80856
2022-11-23T02:42:53.9652156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80857
2022-11-23T02:42:53.9652531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9652705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9653126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9653321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9653688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9653842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9654211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9654396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9654640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9654879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9655279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9655673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9655901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9656128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9656223Z dist init r=0, world=2
2022-11-23T02:42:53.9656332Z dist init r=1, world=2
2022-11-23T02:42:53.9656432Z ok (4.413s)
2022-11-23T02:42:53.9656727Z   test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80939
2022-11-23T02:42:53.9656996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80940
2022-11-23T02:42:53.9657373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9657548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9658085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9658262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9658627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9658797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9659165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9659354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9659594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9659840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9660235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9660625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9660833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9661057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9661170Z dist init r=0, world=2
2022-11-23T02:42:53.9661282Z dist init r=1, world=2
2022-11-23T02:42:53.9661383Z ok (3.910s)
2022-11-23T02:42:53.9661779Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81018
2022-11-23T02:42:53.9662066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81019
2022-11-23T02:42:53.9662446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9662602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9662978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9663168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9663525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9663701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9664069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9664258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9664499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9664738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9665115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9665508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9665735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9666039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9667058Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9667176Z   warnings.warn(
2022-11-23T02:42:53.9668167Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9668277Z   warnings.warn(
2022-11-23T02:42:53.9668895Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9669002Z   warnings.warn(
2022-11-23T02:42:53.9669615Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9669723Z   warnings.warn(
2022-11-23T02:42:53.9669817Z dist init r=0, world=2
2022-11-23T02:42:53.9669923Z dist init r=1, world=2
2022-11-23T02:42:53.9670021Z ok (3.916s)
2022-11-23T02:42:53.9670410Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81097
2022-11-23T02:42:53.9670634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81098
2022-11-23T02:42:53.9671051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9671229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9671610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9671782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9672142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9672314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9672687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9672870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9673116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9673357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9673753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9674128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9674355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9674577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9675344Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:42:53.9675459Z   warnings.warn(
2022-11-23T02:42:53.9676167Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:42:53.9676278Z   warnings.warn(
2022-11-23T02:42:53.9676390Z dist init r=0, world=2
2022-11-23T02:42:53.9676497Z dist init r=1, world=2
2022-11-23T02:42:53.9676580Z ok (4.012s)
2022-11-23T02:42:53.9676973Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81176
2022-11-23T02:42:53.9677199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81177
2022-11-23T02:42:53.9677576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9677747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9678119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9678306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9678664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9678834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9679183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9679373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9679660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9679906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9680303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9680694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9680919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9681140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9682163Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9682277Z   warnings.warn(
2022-11-23T02:42:53.9683267Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9683435Z   warnings.warn(
2022-11-23T02:42:53.9684034Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9684183Z   warnings.warn(
2022-11-23T02:42:53.9684798Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:42:53.9684908Z   warnings.warn(
2022-11-23T02:42:53.9685020Z dist init r=1, world=2
2022-11-23T02:42:53.9685129Z dist init r=0, world=2
2022-11-23T02:42:53.9685227Z ok (4.012s)
2022-11-23T02:42:53.9685617Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81255
2022-11-23T02:42:53.9685825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81256
2022-11-23T02:42:53.9686192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9686367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9686740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9686925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9687280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9687449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9687812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9687999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9688223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9688513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9688916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9689308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9689528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9689749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9690458Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:42:53.9690576Z   warnings.warn(
2022-11-23T02:42:53.9691285Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:42:53.9691395Z   warnings.warn(
2022-11-23T02:42:53.9691489Z dist init r=0, world=2
2022-11-23T02:42:53.9691595Z dist init r=1, world=2
2022-11-23T02:42:53.9691691Z ok (3.912s)
2022-11-23T02:42:53.9692071Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81334
2022-11-23T02:42:53.9692339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81335
2022-11-23T02:42:53.9692706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9692878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9693254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9693424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9693775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9693946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9694309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9694494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9694739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9694975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9695374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9695763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9695975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9696199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9697208Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9697406Z   warnings.warn(
2022-11-23T02:42:53.9698564Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9698679Z   warnings.warn(
2022-11-23T02:42:53.9698791Z dist init r=0, world=2
2022-11-23T02:42:53.9698899Z dist init r=1, world=2
2022-11-23T02:42:53.9699002Z ok (3.912s)
2022-11-23T02:42:53.9699380Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81413
2022-11-23T02:42:53.9699584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81414
2022-11-23T02:42:53.9699956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9700128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9700493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9700676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9701030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9701276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9701646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9701830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9702060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9702298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9702696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9703085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9703308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9703534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9704249Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:42:53.9704356Z   warnings.warn(
2022-11-23T02:42:53.9705056Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:42:53.9705161Z   warnings.warn(
2022-11-23T02:42:53.9705256Z dist init r=1, world=2
2022-11-23T02:42:53.9705360Z dist init r=0, world=2
2022-11-23T02:42:53.9705458Z ok (3.912s)
2022-11-23T02:42:53.9705840Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81492
2022-11-23T02:42:53.9706059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81493
2022-11-23T02:42:53.9706483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9706660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9707032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9707203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9707555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9707724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9708093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9708276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9708520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9708754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9709151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9709539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9709749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9709967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9711038Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9711150Z   warnings.warn(
2022-11-23T02:42:53.9712143Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9712255Z   warnings.warn(
2022-11-23T02:42:53.9712360Z dist init r=1, world=2
2022-11-23T02:42:53.9712467Z dist init r=0, world=2
2022-11-23T02:42:53.9712562Z ok (3.912s)
2022-11-23T02:42:53.9712943Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81571
2022-11-23T02:42:53.9713145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81572
2022-11-23T02:42:53.9713508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9713679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9714054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9714242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9714608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9714772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9715180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9715366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9715591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9715829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9716226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9716624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9716849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9717067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9717770Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:42:53.9717879Z   warnings.warn(
2022-11-23T02:42:53.9718627Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:42:53.9718793Z   warnings.warn(
2022-11-23T02:42:53.9718887Z dist init r=0, world=2
2022-11-23T02:42:53.9718992Z dist init r=1, world=2
2022-11-23T02:42:53.9719087Z ok (3.912s)
2022-11-23T02:42:53.9719372Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9719681Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81650
2022-11-23T02:42:53.9719891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81651
2022-11-23T02:42:53.9720262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9720428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9720787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9720978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9721330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9721503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9721869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9722055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9722291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9722525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9722903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9723299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9723525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9723792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9723904Z dist init r=1, world=2
2022-11-23T02:42:53.9724006Z dist init r=0, world=2
2022-11-23T02:42:53.9724108Z ok (4.112s)
2022-11-23T02:42:53.9724387Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9724681Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81729
2022-11-23T02:42:53.9724892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81730
2022-11-23T02:42:53.9725264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9725434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9725807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9725993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9726352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9726522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9726887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9727053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9727292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9727592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9727984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9728377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9728600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9728821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9728928Z dist init r=0, world=2
2022-11-23T02:42:53.9729033Z dist init r=1, world=2
2022-11-23T02:42:53.9729117Z ok (3.912s)
2022-11-23T02:42:53.9729380Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9729695Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81808
2022-11-23T02:42:53.9729910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81809
2022-11-23T02:42:53.9730279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9730450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9730820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9731005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9731348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9731517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9731890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9732068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9732345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9732586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9732984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9733369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9733593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9733799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9733911Z dist init r=1, world=2
2022-11-23T02:42:53.9734013Z dist init r=0, world=2
2022-11-23T02:42:53.9734110Z ok (4.112s)
2022-11-23T02:42:53.9734374Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:42:53.9734681Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81887
2022-11-23T02:42:53.9734895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81888
2022-11-23T02:42:53.9735264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9735421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9735793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9735977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9736396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9736563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9736928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9737111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9737349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9737570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9738122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9738525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9738750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9738975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9739086Z dist init r=1, world=2
2022-11-23T02:42:53.9739190Z dist init r=0, world=2
2022-11-23T02:42:53.9739286Z ok (4.112s)
2022-11-23T02:42:53.9739626Z   test_state_dict_with_shared_parameters_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81966
2022-11-23T02:42:53.9739842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81967
2022-11-23T02:42:53.9740208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9740379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9740757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9740941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9741364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9741539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9741914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9742082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9742312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9742547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9742946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9743336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9743561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9743787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9743896Z dist init r=1, world=2
2022-11-23T02:42:53.9744003Z dist init r=0, world=2
2022-11-23T02:42:53.9744086Z ok (3.910s)
2022-11-23T02:42:53.9744445Z   test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82045
2022-11-23T02:42:53.9744661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82046
2022-11-23T02:42:53.9745100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9745273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9745655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9745843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9746206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9746360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9746731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9746914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9747152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9747396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9747795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9748185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9748411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9748635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9748731Z dist init r=1, world=2
2022-11-23T02:42:53.9748841Z dist init r=0, world=2
2022-11-23T02:42:53.9748941Z ok (4.011s)
2022-11-23T02:42:53.9749291Z   test_state_dict_with_shared_parameters_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82124
2022-11-23T02:42:53.9749513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82125
2022-11-23T02:42:53.9749926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9750107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9750484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9750654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9751011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9751185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9751553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9751738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9751975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9752215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9752611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9753003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9753211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9753434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9753547Z dist init r=0, world=2
2022-11-23T02:42:53.9753704Z dist init r=1, world=2
2022-11-23T02:42:53.9753802Z ok (4.011s)
2022-11-23T02:42:53.9754104Z   test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82203
2022-11-23T02:42:53.9754322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82204
2022-11-23T02:42:53.9754689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9754845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9755218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9755404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9755758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:42:53.9755929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:42:53.9756303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:42:53.9756489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:42:53.9756734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:42:53.9756955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:42:53.9757352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9757743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:42:53.9757968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:42:53.9758191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:42:53.9759246Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9759368Z   warnings.warn(
2022-11-23T02:42:53.9760358Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:42:53.9760475Z   warnings.warn(
2022-11-23T02:42:53.9760584Z dist init r=1, world=2
2022-11-23T02:42:53.9760690Z dist init r=0, world=2
2022-11-23T02:42:53.9760773Z ok (3.910s)
2022-11-23T02:42:53.9760797Z 
2022-11-23T02:42:53.9761066Z ----------------------------------------------------------------------
2022-11-23T02:42:53.9761185Z Ran 116 tests in 471.994s
2022-11-23T02:42:53.9761206Z 
2022-11-23T02:42:53.9761296Z OK
2022-11-23T02:42:53.9761315Z 
2022-11-23T02:42:53.9761438Z Generating XML reports...
2022-11-23T02:42:53.9761889Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123023501.xml
2022-11-23T02:42:53.9761909Z 
2022-11-23T02:42:53.9762362Z ##[endgroup]
2022-11-23T02:42:53.9762843Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict__e0_b_k9)
2022-11-23T02:42:53.9762919Z 
2022-11-23T02:42:53.9763160Z Running distributed/test_c10d_nccl ... [2022-11-23 02:42:53.824829]
2022-11-23T02:42:53.9763652Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:42:53.825099]
2022-11-23T02:59:56.1267517Z 
2022-11-23T02:59:56.1270619Z Expand the folded group to see the log file of distributed/test_c10d_nccl
2022-11-23T02:59:56.1272131Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_fzn4qwas)
2022-11-23T02:59:56.1275887Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-11-23T02:59:56.1279353Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2022-11-23T02:59:56.1280086Z test_broadcast_coalesced_nccl (__main__.CommTest)
2022-11-23T02:59:56.1287041Z test_nccl_barrier (__main__.CommTest)
2022-11-23T02:59:56.1287368Z test_nccl_barrier_device_ids (__main__.CommTest)
2022-11-23T02:59:56.1289343Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2022-11-23T02:59:56.1289756Z test_nccl_barrier_timeout (__main__.CommTest)
2022-11-23T02:59:56.1290116Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2022-11-23T02:59:56.1290463Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2022-11-23T02:59:56.1290839Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2022-11-23T02:59:56.1291198Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2022-11-23T02:59:56.1291531Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2022-11-23T02:59:56.1291872Z test_nncl_rank_membership (__main__.CommTest)
2022-11-23T02:59:56.1292228Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2022-11-23T02:59:56.1295681Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2022-11-23T02:59:56.1296409Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2022-11-23T02:59:56.1297124Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2022-11-23T02:59:56.1297814Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2022-11-23T02:59:56.1298788Z test_tensor_dtype_complex (__main__.CommTest)
2022-11-23T02:59:56.1299383Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-11-23T02:59:56.1301047Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-11-23T02:59:56.1303069Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:59:56.1303772Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:59:56.1304424Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:59:56.1305099Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:59:56.1305811Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-11-23T02:59:56.1306490Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:59:56.1307127Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:59:56.1324512Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_packed_sequence>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-11-23T02:59:56.1341311Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1342195Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1343023Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1343912Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1344781Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1345612Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1346453Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1347197Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1347971Z test_channels_last_contig (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1348812Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1349661Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1350437Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1351357Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1352327Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1354963Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1355936Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1356860Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1357786Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1358709Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1359673Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1360652Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1361389Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1362012Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1362483Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1362977Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1363434Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1363903Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1364352Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1364785Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1365184Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1365603Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1366038Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1366470Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1366895Z test_failure_recovery (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1367433Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1367924Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1368381Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1368867Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1369372Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1369854Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1370279Z test_fp16 (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1370684Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1371122Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1371516Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1371974Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1372442Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1372850Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1373295Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1373768Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1374258Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1374734Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1375268Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1375696Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1376123Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1376604Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1377100Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1377585Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1378657Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1379085Z test_no_grad (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1379528Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1379935Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1380353Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1380805Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1381257Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1381679Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1382058Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:59:56.1383300Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2022-11-23T02:59:56.1384621Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2022-11-23T02:59:56.1385041Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2022-11-23T02:59:56.1385424Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2022-11-23T02:59:56.1385822Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2022-11-23T02:59:56.1386223Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2022-11-23T02:59:56.1386605Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2022-11-23T02:59:56.1386998Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2022-11-23T02:59:56.1387389Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2022-11-23T02:59:56.1387758Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2022-11-23T02:59:56.1388762Z <unittest.suite.TestSuite tests=[<__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allgather_base>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_reduce_scatter_base>]>
2022-11-23T02:59:56.1389844Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:59:56.1390383Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:59:56.1390915Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:59:56.1391423Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:59:56.1392038Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2022-11-23T02:59:56.1392492Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2022-11-23T02:59:56.1394499Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_nccl_dist_backend_error>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]>
2022-11-23T02:59:56.1396569Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1396947Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1397311Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1397665Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1397991Z test_barrier (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1398333Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1398685Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1399022Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1399368Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1399714Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1400117Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1400490Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1400860Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1401229Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1401601Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1401961Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1402311Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1402645Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1402988Z test_send_recv (__main__.ProcessGroupNCCLTest)
2022-11-23T02:59:56.1403411Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2022-11-23T02:59:56.1403800Z test_common_errors (__main__.RendezvousEnvTest)
2022-11-23T02:59:56.1404123Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:59:56.1404540Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2022-11-23T02:59:56.1404938Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2022-11-23T02:59:56.1405624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1406075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1406646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1407095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1407325Z 
2022-11-23T02:59:56.1407437Z Running tests...
2022-11-23T02:59:56.1407903Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1408410Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1408888Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1409349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82350
2022-11-23T02:59:56.1409795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82351
2022-11-23T02:59:56.1410380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1410825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1411396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1411862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1412423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1412863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1413430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1413875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1414305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1414771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1415764Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:59:56.1416367Z   warnings.warn(
2022-11-23T02:59:56.1417269Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:59:56.1418553Z   warnings.warn(
2022-11-23T02:59:56.1418918Z ok (6.667s)
2022-11-23T02:59:56.1419053Z 
2022-11-23T02:59:56.1419340Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1419666Z Ran 1 test in 6.667s
2022-11-23T02:59:56.1419825Z 
2022-11-23T02:59:56.1419919Z OK
2022-11-23T02:59:56.1420051Z 
2022-11-23T02:59:56.1420157Z Generating XML reports...
2022-11-23T02:59:56.1420697Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024257.xml
2022-11-23T02:59:56.1421346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1421799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1422350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1422816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1423045Z 
2022-11-23T02:59:56.1423156Z Running tests...
2022-11-23T02:59:56.1423537Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1424056Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1424534Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1424989Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82467
2022-11-23T02:59:56.1425412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82468
2022-11-23T02:59:56.1426183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1426627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1427183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1427647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1428220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1428659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1429206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1429662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1430098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1430563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1430887Z ok (6.638s)
2022-11-23T02:59:56.1431034Z 
2022-11-23T02:59:56.1431306Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1431631Z Ran 1 test in 6.638s
2022-11-23T02:59:56.1431791Z 
2022-11-23T02:59:56.1431868Z OK
2022-11-23T02:59:56.1431999Z 
2022-11-23T02:59:56.1432125Z Generating XML reports...
2022-11-23T02:59:56.1432660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024306.xml
2022-11-23T02:59:56.1433293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1433730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1434293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1434755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1434979Z 
2022-11-23T02:59:56.1435071Z Running tests...
2022-11-23T02:59:56.1435534Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1436066Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1436517Z   test_nccl_barrier (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1436937Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82584
2022-11-23T02:59:56.1437376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82585
2022-11-23T02:59:56.1437977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1438403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1438971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1439435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1440007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1440428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1440993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1441449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1441862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1442323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1442759Z skip: Need at least 4 CUDA devices (3.910s)
2022-11-23T02:59:56.1442947Z 
2022-11-23T02:59:56.1443211Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1443517Z Ran 1 test in 3.910s
2022-11-23T02:59:56.1443677Z 
2022-11-23T02:59:56.1443785Z OK (skipped=1)
2022-11-23T02:59:56.1443941Z 
2022-11-23T02:59:56.1444065Z Generating XML reports...
2022-11-23T02:59:56.1444582Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024315.xml
2022-11-23T02:59:56.1445231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1445677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1446240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1446702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1446933Z 
2022-11-23T02:59:56.1447043Z Running tests...
2022-11-23T02:59:56.1447422Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1447941Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1448411Z   test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1448849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82687
2022-11-23T02:59:56.1449289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82688
2022-11-23T02:59:56.1449881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1450321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1450871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1451337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1451903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1452374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1452950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1453403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1453832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1454298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1454774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1455249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1455896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1456562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1456952Z ok (5.382s)
2022-11-23T02:59:56.1457098Z 
2022-11-23T02:59:56.1457360Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1457667Z Ran 1 test in 5.382s
2022-11-23T02:59:56.1457828Z 
2022-11-23T02:59:56.1458268Z OK
2022-11-23T02:59:56.1458412Z 
2022-11-23T02:59:56.1458538Z Generating XML reports...
2022-11-23T02:59:56.1459078Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024321.xml
2022-11-23T02:59:56.1459714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1460239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1460807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1461258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1461485Z 
2022-11-23T02:59:56.1461593Z Running tests...
2022-11-23T02:59:56.1461989Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1462508Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1462982Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1463455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82803
2022-11-23T02:59:56.1463896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82804
2022-11-23T02:59:56.1464476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1464923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1465493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1465954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1466507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1466951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1467514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1467975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1468385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1468871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1469348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1469870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1470531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1471214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1471601Z ok (3.916s)
2022-11-23T02:59:56.1471729Z 
2022-11-23T02:59:56.1471992Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1472315Z Ran 1 test in 3.916s
2022-11-23T02:59:56.1472474Z 
2022-11-23T02:59:56.1472565Z OK
2022-11-23T02:59:56.1472702Z 
2022-11-23T02:59:56.1472807Z Generating XML reports...
2022-11-23T02:59:56.1473344Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024329.xml
2022-11-23T02:59:56.1473998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1474442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1474984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1475445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1475671Z 
2022-11-23T02:59:56.1475780Z Running tests...
2022-11-23T02:59:56.1476162Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1476677Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1477143Z   test_nccl_barrier_timeout (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1477651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82910
2022-11-23T02:59:56.1478078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82911
2022-11-23T02:59:56.1478678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1479121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1479714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1480179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1480746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1481187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1481732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1482190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1482625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1483087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1483453Z skip: Need at least 4 CUDA devices (3.926s)
2022-11-23T02:59:56.1483642Z 
2022-11-23T02:59:56.1483907Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1484231Z Ran 1 test in 3.926s
2022-11-23T02:59:56.1484389Z 
2022-11-23T02:59:56.1484481Z OK (skipped=1)
2022-11-23T02:59:56.1484631Z 
2022-11-23T02:59:56.1484754Z Generating XML reports...
2022-11-23T02:59:56.1485283Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024335.xml
2022-11-23T02:59:56.1485938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1486364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1486980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1487447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1487674Z 
2022-11-23T02:59:56.1487765Z Running tests...
2022-11-23T02:59:56.1488163Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1488682Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1489157Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1489599Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83013
2022-11-23T02:59:56.1490040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83014
2022-11-23T02:59:56.1490637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1491064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1491626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1492088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1492653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1493068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1493631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1494140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1494552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1495017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1495402Z skip: Need at least 4 CUDA devices (4.040s)
2022-11-23T02:59:56.1495595Z 
2022-11-23T02:59:56.1495861Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1496173Z Ran 1 test in 4.040s
2022-11-23T02:59:56.1496331Z 
2022-11-23T02:59:56.1496440Z OK (skipped=1)
2022-11-23T02:59:56.1496593Z 
2022-11-23T02:59:56.1496718Z Generating XML reports...
2022-11-23T02:59:56.1497234Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024342.xml
2022-11-23T02:59:56.1498203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1498673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1499247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1499693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1499920Z 
2022-11-23T02:59:56.1500028Z Running tests...
2022-11-23T02:59:56.1500426Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1500927Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1501425Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1501899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83116
2022-11-23T02:59:56.1502341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83117
2022-11-23T02:59:56.1502923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1503364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1504005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1504475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1505030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1505466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1506028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1506466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1506894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1507362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1507745Z skip: Need at least 4 CUDA devices (3.927s)
2022-11-23T02:59:56.1507917Z 
2022-11-23T02:59:56.1508185Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1508508Z Ran 1 test in 3.928s
2022-11-23T02:59:56.1508665Z 
2022-11-23T02:59:56.1508773Z OK (skipped=1)
2022-11-23T02:59:56.1508926Z 
2022-11-23T02:59:56.1509032Z Generating XML reports...
2022-11-23T02:59:56.1509565Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024348.xml
2022-11-23T02:59:56.1510214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1510654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1511201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1511732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1511953Z 
2022-11-23T02:59:56.1512061Z Running tests...
2022-11-23T02:59:56.1512448Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1512972Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1513450Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1513918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83219
2022-11-23T02:59:56.1514342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83220
2022-11-23T02:59:56.1514935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1515377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1515929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1516390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1516962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1517398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1517943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1518399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1518827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1519296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1519762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1520253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1520949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1521624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1522148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:59:56.1522631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:59:56.1523271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1523927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1524319Z ok (5.502s)
2022-11-23T02:59:56.1524467Z 
2022-11-23T02:59:56.1524728Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1525047Z Ran 1 test in 5.502s
2022-11-23T02:59:56.1525191Z 
2022-11-23T02:59:56.1525283Z OK
2022-11-23T02:59:56.1525415Z 
2022-11-23T02:59:56.1525539Z Generating XML reports...
2022-11-23T02:59:56.1526074Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024354.xml
2022-11-23T02:59:56.1526713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1527154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1527719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1528181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1528440Z 
2022-11-23T02:59:56.1528548Z Running tests...
2022-11-23T02:59:56.1528942Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1529466Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1529930Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1530390Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83350
2022-11-23T02:59:56.1530834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83351
2022-11-23T02:59:56.1531433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1531861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1532423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1532892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1533441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1533880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1534439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1534895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1535303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1535782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1536258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1536736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1537364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1538299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:59:56.1538972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1539477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:59:56.1540118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1540790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1541177Z ok (5.520s)
2022-11-23T02:59:56.1541328Z 
2022-11-23T02:59:56.1541576Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1541901Z Ran 1 test in 5.520s
2022-11-23T02:59:56.1542060Z 
2022-11-23T02:59:56.1542152Z OK
2022-11-23T02:59:56.1542283Z 
2022-11-23T02:59:56.1542391Z Generating XML reports...
2022-11-23T02:59:56.1542926Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024402.xml
2022-11-23T02:59:56.1543581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1544026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1544574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1545032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1545258Z 
2022-11-23T02:59:56.1545368Z Running tests...
2022-11-23T02:59:56.1545845Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1546351Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1546833Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1547296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83472
2022-11-23T02:59:56.1547722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83473
2022-11-23T02:59:56.1548312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1548755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1549319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1549762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1550333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1550773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1551324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1551783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1552215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1552693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1553152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1553630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1554283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1554809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:59:56.1555473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1556004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:59:56.1556643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1557303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1557692Z ok (5.541s)
2022-11-23T02:59:56.1557840Z 
2022-11-23T02:59:56.1558104Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1558435Z Ran 1 test in 5.541s
2022-11-23T02:59:56.1558575Z 
2022-11-23T02:59:56.1558669Z OK
2022-11-23T02:59:56.1558802Z 
2022-11-23T02:59:56.1558926Z Generating XML reports...
2022-11-23T02:59:56.1559463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024410.xml
2022-11-23T02:59:56.1560104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1560544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1561104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1561564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1561787Z 
2022-11-23T02:59:56.1561879Z Running tests...
2022-11-23T02:59:56.1562278Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1562858Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1563303Z   test_nncl_rank_membership (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1563755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83594
2022-11-23T02:59:56.1564202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83595
2022-11-23T02:59:56.1564797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1565220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1565783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1566241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1566808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1567229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1567793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1568256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1568668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1569147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1569625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1570099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1570728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1571257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:59:56.1571894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1572466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:59:56.1573099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1573771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1574156Z ok (3.951s)
2022-11-23T02:59:56.1574302Z 
2022-11-23T02:59:56.1574549Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1574871Z Ran 1 test in 3.951s
2022-11-23T02:59:56.1575030Z 
2022-11-23T02:59:56.1575125Z OK
2022-11-23T02:59:56.1575258Z 
2022-11-23T02:59:56.1575381Z Generating XML reports...
2022-11-23T02:59:56.1575891Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024418.xml
2022-11-23T02:59:56.1576545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1576990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1577538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1578355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1578589Z 
2022-11-23T02:59:56.1578698Z Running tests...
2022-11-23T02:59:56.1579106Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1579648Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1580222Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1580699Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83703
2022-11-23T02:59:56.1581129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83704
2022-11-23T02:59:56.1581733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1582177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1582740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1583187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1583755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1584191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1584742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1585201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1585637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1586122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1586585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1587056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1587699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1588223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:59:56.1588849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1589440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:59:56.1590088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1590763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1591131Z ok (6.643s)
2022-11-23T02:59:56.1591278Z 
2022-11-23T02:59:56.1591547Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1591870Z Ran 1 test in 6.643s
2022-11-23T02:59:56.1592028Z 
2022-11-23T02:59:56.1592102Z OK
2022-11-23T02:59:56.1592234Z 
2022-11-23T02:59:56.1592359Z Generating XML reports...
2022-11-23T02:59:56.1592899Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024424.xml
2022-11-23T02:59:56.1593550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1593979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1594543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1595001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1595226Z 
2022-11-23T02:59:56.1595318Z Running tests...
2022-11-23T02:59:56.1595767Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1596287Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1596780Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1597293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83824
2022-11-23T02:59:56.1597733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83825
2022-11-23T02:59:56.1598334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1598759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1599321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1599783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1600351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1600771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1601336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1601796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1602209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1602691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1603167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1603645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1604274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1604953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1605478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:59:56.1605965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:59:56.1606635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1607312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1607698Z ok (5.635s)
2022-11-23T02:59:56.1607846Z 
2022-11-23T02:59:56.1608113Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1608419Z Ran 1 test in 5.635s
2022-11-23T02:59:56.1608576Z 
2022-11-23T02:59:56.1608669Z OK
2022-11-23T02:59:56.1608802Z 
2022-11-23T02:59:56.1608927Z Generating XML reports...
2022-11-23T02:59:56.1609444Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024433.xml
2022-11-23T02:59:56.1610098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1610537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1611087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1611547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1611770Z 
2022-11-23T02:59:56.1611879Z Running tests...
2022-11-23T02:59:56.1612275Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1612782Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1613274Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1613747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83946
2022-11-23T02:59:56.1614242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83947
2022-11-23T02:59:56.1614827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1615269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1615839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1616283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1616849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1617285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1617844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1618647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1619078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1619541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1619912Z skip: Need at least 4 CUDA devices (3.933s)
2022-11-23T02:59:56.1620101Z 
2022-11-23T02:59:56.1620374Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1620698Z Ran 1 test in 3.934s
2022-11-23T02:59:56.1620856Z 
2022-11-23T02:59:56.1620963Z OK (skipped=1)
2022-11-23T02:59:56.1621099Z 
2022-11-23T02:59:56.1621222Z Generating XML reports...
2022-11-23T02:59:56.1621754Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024441.xml
2022-11-23T02:59:56.1622403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1622833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1623400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1623933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1624164Z 
2022-11-23T02:59:56.1624272Z Running tests...
2022-11-23T02:59:56.1624654Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1625173Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1625650Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1626099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84049
2022-11-23T02:59:56.1626539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84050
2022-11-23T02:59:56.1627132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1627580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1628133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1628594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1629159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1629598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1630145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1630603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1631033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1631567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1632038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1632520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1633162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1633823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1634212Z ok (5.555s)
2022-11-23T02:59:56.1634357Z 
2022-11-23T02:59:56.1634621Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1634928Z Ran 1 test in 5.555s
2022-11-23T02:59:56.1635090Z 
2022-11-23T02:59:56.1635180Z OK
2022-11-23T02:59:56.1635312Z 
2022-11-23T02:59:56.1635440Z Generating XML reports...
2022-11-23T02:59:56.1635976Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024447.xml
2022-11-23T02:59:56.1636621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1637062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1637626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1638074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1638299Z 
2022-11-23T02:59:56.1638407Z Running tests...
2022-11-23T02:59:56.1638804Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1639321Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1639786Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1640250Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84165
2022-11-23T02:59:56.1640693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84166
2022-11-23T02:59:56.1641322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1641772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1642337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1642800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1643350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1643785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1644353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1644810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1645225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1645709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1646185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1646646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1647295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1647820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:59:56.1648510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1649016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:59:56.1649654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1650324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:59:56.1650710Z ok (5.542s)
2022-11-23T02:59:56.1650839Z 
2022-11-23T02:59:56.1651105Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1651428Z Ran 1 test in 5.542s
2022-11-23T02:59:56.1651585Z 
2022-11-23T02:59:56.1651677Z OK
2022-11-23T02:59:56.1651808Z 
2022-11-23T02:59:56.1651913Z Generating XML reports...
2022-11-23T02:59:56.1652446Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024455.xml
2022-11-23T02:59:56.1653105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1653548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1654100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1654561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1654787Z 
2022-11-23T02:59:56.1654895Z Running tests...
2022-11-23T02:59:56.1655279Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1655798Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1656263Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1656714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84285
2022-11-23T02:59:56.1657142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84286
2022-11-23T02:59:56.1657733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1658483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1659048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1659509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1660076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1660515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1661064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1661521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1661951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1662421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1662898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1663374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1664020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1664684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1665069Z ok (6.564s)
2022-11-23T02:59:56.1665221Z 
2022-11-23T02:59:56.1665565Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1665889Z Ran 1 test in 6.565s
2022-11-23T02:59:56.1666032Z 
2022-11-23T02:59:56.1666125Z OK
2022-11-23T02:59:56.1666256Z 
2022-11-23T02:59:56.1666381Z Generating XML reports...
2022-11-23T02:59:56.1666916Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024503.xml
2022-11-23T02:59:56.1667550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1667992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1668558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1669018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1669225Z 
2022-11-23T02:59:56.1669336Z Running tests...
2022-11-23T02:59:56.1669733Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1670262Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1670710Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1671158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84402
2022-11-23T02:59:56.1671603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84403
2022-11-23T02:59:56.1672197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1672623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1673185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1673644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1674198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1674633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1675249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1675713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1676126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1676606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1677083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1677554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1678182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1678870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1679950Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:59:56.1680587Z   warnings.warn(
2022-11-23T02:59:56.1681435Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:59:56.1682045Z   warnings.warn(
2022-11-23T02:59:56.1682897Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:59:56.1683564Z   warnings.warn(
2022-11-23T02:59:56.1684402Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:59:56.1685004Z   warnings.warn(
2022-11-23T02:59:56.1685239Z ok (5.330s)
2022-11-23T02:59:56.1685385Z 
2022-11-23T02:59:56.1685650Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1685958Z Ran 1 test in 5.330s
2022-11-23T02:59:56.1686116Z 
2022-11-23T02:59:56.1686208Z OK
2022-11-23T02:59:56.1686339Z 
2022-11-23T02:59:56.1686462Z Generating XML reports...
2022-11-23T02:59:56.1686984Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024512.xml
2022-11-23T02:59:56.1687639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1688088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1688651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1689096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1689321Z 
2022-11-23T02:59:56.1689429Z Running tests...
2022-11-23T02:59:56.1689830Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1690335Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1690813Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1691276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84511
2022-11-23T02:59:56.1691720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84512
2022-11-23T02:59:56.1692348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1692792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1693360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1693803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1694370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1694805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1695362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1695804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1696235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1696703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1697164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1697650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1698539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1699219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1700206Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1700918Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1701757Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1702457Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1702782Z ok (6.640s)
2022-11-23T02:59:56.1702911Z 
2022-11-23T02:59:56.1703174Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1703499Z Ran 1 test in 6.640s
2022-11-23T02:59:56.1703664Z 
2022-11-23T02:59:56.1703756Z OK
2022-11-23T02:59:56.1703887Z 
2022-11-23T02:59:56.1703994Z Generating XML reports...
2022-11-23T02:59:56.1704544Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024520.xml
2022-11-23T02:59:56.1705208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1705654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1706205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1706664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1706890Z 
2022-11-23T02:59:56.1706999Z Running tests...
2022-11-23T02:59:56.1707381Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1707905Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1708388Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1708852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84626
2022-11-23T02:59:56.1709335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84627
2022-11-23T02:59:56.1709937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1710383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1710938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1711407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1711979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1712424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1712972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1713430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1713865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1714313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1714791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1715278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1715928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1716588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1717563Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1718276Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1719122Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1719824Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1720128Z ok (6.633s)
2022-11-23T02:59:56.1720275Z 
2022-11-23T02:59:56.1720540Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1720867Z Ran 1 test in 6.633s
2022-11-23T02:59:56.1721025Z 
2022-11-23T02:59:56.1721101Z OK
2022-11-23T02:59:56.1721233Z 
2022-11-23T02:59:56.1721360Z Generating XML reports...
2022-11-23T02:59:56.1721918Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024529.xml
2022-11-23T02:59:56.1722566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1723008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1723575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1724040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1724265Z 
2022-11-23T02:59:56.1724356Z Running tests...
2022-11-23T02:59:56.1724752Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1725275Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1725752Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1726242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84741
2022-11-23T02:59:56.1726689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84742
2022-11-23T02:59:56.1727286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1727711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1728276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1728734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1729301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1729720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1730285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1730742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1731153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1731617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1732087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1732574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1733203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1733928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1734844Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1735552Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1736375Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1737079Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1737410Z ok (6.739s)
2022-11-23T02:59:56.1737557Z 
2022-11-23T02:59:56.1737819Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1738315Z Ran 1 test in 6.739s
2022-11-23T02:59:56.1738479Z 
2022-11-23T02:59:56.1738572Z OK
2022-11-23T02:59:56.1738705Z 
2022-11-23T02:59:56.1738836Z Generating XML reports...
2022-11-23T02:59:56.1739373Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024538.xml
2022-11-23T02:59:56.1740043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1740492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1741058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1741504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1741728Z 
2022-11-23T02:59:56.1741841Z Running tests...
2022-11-23T02:59:56.1742238Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1742743Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1743302Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1743790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84856
2022-11-23T02:59:56.1744231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84857
2022-11-23T02:59:56.1744813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1745251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1745813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1746275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1746831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1747264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1747824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1748264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1748694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1749159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1749638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1750104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1750831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1751516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1752433Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1753128Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1753966Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1754672Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1755508Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1756193Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1757021Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1757715Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1758542Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1759248Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1760132Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1760839Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1761672Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1762366Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1763190Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1763889Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1764212Z ok (6.596s)
2022-11-23T02:59:56.1764359Z 
2022-11-23T02:59:56.1764623Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1764932Z Ran 1 test in 6.596s
2022-11-23T02:59:56.1765091Z 
2022-11-23T02:59:56.1765182Z OK
2022-11-23T02:59:56.1765313Z 
2022-11-23T02:59:56.1765437Z Generating XML reports...
2022-11-23T02:59:56.1765971Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024547.xml
2022-11-23T02:59:56.1766629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1767124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1767692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1768147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1768372Z 
2022-11-23T02:59:56.1768481Z Running tests...
2022-11-23T02:59:56.1768880Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1769396Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1769862Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1770332Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84971
2022-11-23T02:59:56.1770781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84972
2022-11-23T02:59:56.1771369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1771811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1772377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1772844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1773398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1773830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1774391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1774831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1775263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1775736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1776216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1776730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1777391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1778300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1779228Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1779958Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1780798Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1781500Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1782340Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1783023Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1783858Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1784655Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1784981Z ok (6.643s)
2022-11-23T02:59:56.1785129Z 
2022-11-23T02:59:56.1785376Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1785696Z Ran 1 test in 6.643s
2022-11-23T02:59:56.1785855Z 
2022-11-23T02:59:56.1785948Z OK
2022-11-23T02:59:56.1786078Z 
2022-11-23T02:59:56.1786205Z Generating XML reports...
2022-11-23T02:59:56.1786739Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024556.xml
2022-11-23T02:59:56.1787411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1787849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1788400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1788859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1789086Z 
2022-11-23T02:59:56.1789197Z Running tests...
2022-11-23T02:59:56.1789595Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1790099Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1790593Z   test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1791068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85086
2022-11-23T02:59:56.1791492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85087
2022-11-23T02:59:56.1792088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1792533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1793100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1793613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1794195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1794633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1795192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1795635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1796067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1796531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1796995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1797478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1798124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1798804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1799702Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1800412Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1801305Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1802010Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1802843Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1803520Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1804351Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1805052Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1805374Z ok (6.608s)
2022-11-23T02:59:56.1805502Z 
2022-11-23T02:59:56.1805773Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1806100Z Ran 1 test in 6.608s
2022-11-23T02:59:56.1806256Z 
2022-11-23T02:59:56.1806348Z OK
2022-11-23T02:59:56.1806482Z 
2022-11-23T02:59:56.1806592Z Generating XML reports...
2022-11-23T02:59:56.1807143Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024605.xml
2022-11-23T02:59:56.1807802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1808244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1808792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1809257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1809483Z 
2022-11-23T02:59:56.1809593Z Running tests...
2022-11-23T02:59:56.1810023Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1810558Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1811033Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1811488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85201
2022-11-23T02:59:56.1811906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85202
2022-11-23T02:59:56.1812497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1812938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1813490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1813952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1814527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1814965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1815508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1815966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1816396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1816843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1817322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.1817856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.1818760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1819426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.1820341Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1821050Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1821888Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:59:56.1849286Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:59:56.1849703Z ok (6.740s)
2022-11-23T02:59:56.1849864Z 
2022-11-23T02:59:56.1850185Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1850495Z Ran 1 test in 6.740s
2022-11-23T02:59:56.1850658Z 
2022-11-23T02:59:56.1850751Z OK
2022-11-23T02:59:56.1850882Z 
2022-11-23T02:59:56.1851016Z Generating XML reports...
2022-11-23T02:59:56.1851568Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024614.xml
2022-11-23T02:59:56.1852244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1852692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1853272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1853724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1853951Z 
2022-11-23T02:59:56.1854221Z Running tests...
2022-11-23T02:59:56.1854643Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1855146Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1855669Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1856175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85318
2022-11-23T02:59:56.1856621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85319
2022-11-23T02:59:56.1857204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1857658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1858587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1859043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1859617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1860053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1860618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1861062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1861496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1861962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1862539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_2tzciw
2022-11-23T02:59:56.1863074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_2tzciw/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1863599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfwlq8un
2022-11-23T02:59:56.1864130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfwlq8un/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1864624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1865108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1865455Z ok (7.040s)
2022-11-23T02:59:56.1865603Z 
2022-11-23T02:59:56.1865873Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1866182Z Ran 1 test in 7.041s
2022-11-23T02:59:56.1866346Z 
2022-11-23T02:59:56.1866442Z OK
2022-11-23T02:59:56.1866575Z 
2022-11-23T02:59:56.1866700Z Generating XML reports...
2022-11-23T02:59:56.1867298Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024623.xml
2022-11-23T02:59:56.1868022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1868470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1869038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1869486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1869713Z 
2022-11-23T02:59:56.1869824Z Running tests...
2022-11-23T02:59:56.1870223Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1870729Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1871277Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1871880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85439
2022-11-23T02:59:56.1872337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85440
2022-11-23T02:59:56.1872924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1873367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1873932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1874378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1874741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1874921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1875295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1875489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1875717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1875943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1876198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q94b5wn
2022-11-23T02:59:56.1876468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q94b5wn/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1876701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvao_cb5i
2022-11-23T02:59:56.1877021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvao_cb5i/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1877257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1877493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1877597Z ok (7.141s)
2022-11-23T02:59:56.1877616Z 
2022-11-23T02:59:56.1877887Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1878001Z Ran 1 test in 7.142s
2022-11-23T02:59:56.1878020Z 
2022-11-23T02:59:56.1878113Z OK
2022-11-23T02:59:56.1878132Z 
2022-11-23T02:59:56.1878237Z Generating XML reports...
2022-11-23T02:59:56.1878698Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024632.xml
2022-11-23T02:59:56.1879068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1879247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1879673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1879868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1879887Z 
2022-11-23T02:59:56.1879998Z Running tests...
2022-11-23T02:59:56.1880262Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1880576Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1880839Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1881058Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85560
2022-11-23T02:59:56.1881276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85561
2022-11-23T02:59:56.1881648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1881821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1882233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1882410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1882786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1882955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1883328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1883517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1883746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1883976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1884231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2kf_oijg
2022-11-23T02:59:56.1884501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2kf_oijg/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1884756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5pkz56k
2022-11-23T02:59:56.1885019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5pkz56k/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1885232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1885467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1885571Z ok (7.154s)
2022-11-23T02:59:56.1885591Z 
2022-11-23T02:59:56.1885902Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1886016Z Ran 1 test in 7.155s
2022-11-23T02:59:56.1886035Z 
2022-11-23T02:59:56.1886129Z OK
2022-11-23T02:59:56.1886147Z 
2022-11-23T02:59:56.1886274Z Generating XML reports...
2022-11-23T02:59:56.1886738Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024642.xml
2022-11-23T02:59:56.1887087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1887263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1887641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1887833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1887852Z 
2022-11-23T02:59:56.1887963Z Running tests...
2022-11-23T02:59:56.1888225Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1888541Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1888843Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1889062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85681
2022-11-23T02:59:56.1889258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85682
2022-11-23T02:59:56.1889629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1889801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1890178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1890367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1890738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1890910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1891322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1891495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1891722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1891948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1892203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qlxpk0r
2022-11-23T02:59:56.1892473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qlxpk0r/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1892727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8voofblb
2022-11-23T02:59:56.1892993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8voofblb/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1893228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1893455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1893540Z ok (7.154s)
2022-11-23T02:59:56.1893559Z 
2022-11-23T02:59:56.1893824Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1893937Z Ran 1 test in 7.154s
2022-11-23T02:59:56.1893956Z 
2022-11-23T02:59:56.1894050Z OK
2022-11-23T02:59:56.1894068Z 
2022-11-23T02:59:56.1894192Z Generating XML reports...
2022-11-23T02:59:56.1894646Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024651.xml
2022-11-23T02:59:56.1895082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1895259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1895672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1895865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1895884Z 
2022-11-23T02:59:56.1895994Z Running tests...
2022-11-23T02:59:56.1896257Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1896573Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1896851Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1897068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85802
2022-11-23T02:59:56.1897288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85803
2022-11-23T02:59:56.1897657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1897814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1898457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1898648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1899015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1899184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1899557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1899746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1899981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1900605Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.1900819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1901348Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.1901612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4paaluc
2022-11-23T02:59:56.1901884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4paaluc/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1902142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnuyjm5tz
2022-11-23T02:59:56.1902410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnuyjm5tz/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1902513Z ok (6.618s)
2022-11-23T02:59:56.1902533Z 
2022-11-23T02:59:56.1902805Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1902919Z Ran 1 test in 6.619s
2022-11-23T02:59:56.1902938Z 
2022-11-23T02:59:56.1903031Z OK
2022-11-23T02:59:56.1903050Z 
2022-11-23T02:59:56.1903155Z Generating XML reports...
2022-11-23T02:59:56.1903617Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024701.xml
2022-11-23T02:59:56.1904049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1904225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1904605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1904796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1904816Z 
2022-11-23T02:59:56.1904927Z Running tests...
2022-11-23T02:59:56.1905193Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1905483Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1905760Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1905979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85923
2022-11-23T02:59:56.1906199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85924
2022-11-23T02:59:56.1906566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1906746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1907123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1907313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1907677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1907832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1908201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1908391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1908619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1909204Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.1909432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1909961Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.1910224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwosjbkac
2022-11-23T02:59:56.1910494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwosjbkac/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1910750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6cugueh
2022-11-23T02:59:56.1910998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6cugueh/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1911102Z ok (6.565s)
2022-11-23T02:59:56.1911122Z 
2022-11-23T02:59:56.1911390Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1911503Z Ran 1 test in 6.565s
2022-11-23T02:59:56.1911522Z 
2022-11-23T02:59:56.1911616Z OK
2022-11-23T02:59:56.1911634Z 
2022-11-23T02:59:56.1911758Z Generating XML reports...
2022-11-23T02:59:56.1912217Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024710.xml
2022-11-23T02:59:56.1912632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1912806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1913170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1913359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1913378Z 
2022-11-23T02:59:56.1913488Z Running tests...
2022-11-23T02:59:56.1913750Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1914060Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1914335Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1914554Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86044
2022-11-23T02:59:56.1914779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86045
2022-11-23T02:59:56.1915127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1915308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1915682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1915871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1916237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1916411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1916780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1916970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1917197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1917448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1917709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7udlkh1
2022-11-23T02:59:56.1917976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7udlkh1/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1918228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppn6ccfk
2022-11-23T02:59:56.1918496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppn6ccfk/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1918601Z ok (6.655s)
2022-11-23T02:59:56.1918620Z 
2022-11-23T02:59:56.1918885Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1919003Z Ran 1 test in 6.655s
2022-11-23T02:59:56.1919022Z 
2022-11-23T02:59:56.1919097Z OK
2022-11-23T02:59:56.1919115Z 
2022-11-23T02:59:56.1919239Z Generating XML reports...
2022-11-23T02:59:56.1919702Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024719.xml
2022-11-23T02:59:56.1920069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1920243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1920617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1920809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1920828Z 
2022-11-23T02:59:56.1920939Z Running tests...
2022-11-23T02:59:56.1921204Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1921547Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1921835Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1922053Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86165
2022-11-23T02:59:56.1922268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86166
2022-11-23T02:59:56.1922638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1922811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1923186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1923377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1923719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1923896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1924271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1924457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1924684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1924903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1925159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6kx_k_v
2022-11-23T02:59:56.1925427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6kx_k_v/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1925678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ese5nbr
2022-11-23T02:59:56.1925929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ese5nbr/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1926033Z ok (6.654s)
2022-11-23T02:59:56.1926053Z 
2022-11-23T02:59:56.1926365Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1926481Z Ran 1 test in 6.654s
2022-11-23T02:59:56.1926500Z 
2022-11-23T02:59:56.1926593Z OK
2022-11-23T02:59:56.1926612Z 
2022-11-23T02:59:56.1926737Z Generating XML reports...
2022-11-23T02:59:56.1927199Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024728.xml
2022-11-23T02:59:56.1927570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1927724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1928099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1928293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1928312Z 
2022-11-23T02:59:56.1928423Z Running tests...
2022-11-23T02:59:56.1928687Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1928999Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1929268Z   test_channels_last_contig (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1929485Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86286
2022-11-23T02:59:56.1929702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86287
2022-11-23T02:59:56.1930053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1930227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1930643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1930815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1931194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1931383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1931756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1931947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1932157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1932384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1932491Z ok (6.598s)
2022-11-23T02:59:56.1932510Z 
2022-11-23T02:59:56.1932775Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1932887Z Ran 1 test in 6.598s
2022-11-23T02:59:56.1932906Z 
2022-11-23T02:59:56.1933000Z OK
2022-11-23T02:59:56.1933018Z 
2022-11-23T02:59:56.1933145Z Generating XML reports...
2022-11-23T02:59:56.1933604Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024737.xml
2022-11-23T02:59:56.1933972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1934126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1934496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1934682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1934704Z 
2022-11-23T02:59:56.1934814Z Running tests...
2022-11-23T02:59:56.1935073Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1935384Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1935652Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1936025Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1936222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86403
2022-11-23T02:59:56.1936436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86404
2022-11-23T02:59:56.1936802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1936976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1937347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1937539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1938129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1938309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1938682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1938851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1939079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1939305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1939561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykt9qbj0
2022-11-23T02:59:56.1939906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykt9qbj0/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1940157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd2qcyy7
2022-11-23T02:59:56.1940426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd2qcyy7/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1940531Z ok (6.004s)
2022-11-23T02:59:56.1940551Z 
2022-11-23T02:59:56.1940799Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1940910Z Ran 1 test in 6.005s
2022-11-23T02:59:56.1940929Z 
2022-11-23T02:59:56.1941026Z OK
2022-11-23T02:59:56.1941044Z 
2022-11-23T02:59:56.1941168Z Generating XML reports...
2022-11-23T02:59:56.1941627Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024746.xml
2022-11-23T02:59:56.1941994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1942174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1942546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1942739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1942758Z 
2022-11-23T02:59:56.1942849Z Running tests...
2022-11-23T02:59:56.1943110Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1943422Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1943662Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1943929Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1944145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86523
2022-11-23T02:59:56.1944364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86524
2022-11-23T02:59:56.1944733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1944944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1945338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1945528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1945888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1946064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1946435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1946627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1946855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1947065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1947316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5si3gov3
2022-11-23T02:59:56.1947582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5si3gov3/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1947831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b14wsac
2022-11-23T02:59:56.1948094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b14wsac/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1948198Z ok (6.045s)
2022-11-23T02:59:56.1948217Z 
2022-11-23T02:59:56.1948490Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1948650Z Ran 1 test in 6.045s
2022-11-23T02:59:56.1948669Z 
2022-11-23T02:59:56.1948762Z OK
2022-11-23T02:59:56.1948780Z 
2022-11-23T02:59:56.1948886Z Generating XML reports...
2022-11-23T02:59:56.1949353Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024754.xml
2022-11-23T02:59:56.1949720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1949893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1950267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1950458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1950477Z 
2022-11-23T02:59:56.1950587Z Running tests...
2022-11-23T02:59:56.1950847Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1951143Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1951382Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1951635Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1951849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86643
2022-11-23T02:59:56.1952066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86644
2022-11-23T02:59:56.1952435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1952611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1952988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1953176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1953527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1953699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1954111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1954304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1954532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1954758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1955017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68tadbc_
2022-11-23T02:59:56.1955283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68tadbc_/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1955517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5vn0mwd
2022-11-23T02:59:56.1955783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5vn0mwd/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1956022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1956259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1956487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1956721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1957633Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:59:56.1957807Z   warnings.warn(
2022-11-23T02:59:56.1958717Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:59:56.1958831Z   warnings.warn(
2022-11-23T02:59:56.1959061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1959267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1959496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1959725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1959962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1960184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1960408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1960635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1960737Z ok (6.126s)
2022-11-23T02:59:56.1960757Z 
2022-11-23T02:59:56.1961006Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1961118Z Ran 1 test in 6.126s
2022-11-23T02:59:56.1961138Z 
2022-11-23T02:59:56.1961231Z OK
2022-11-23T02:59:56.1961250Z 
2022-11-23T02:59:56.1961376Z Generating XML reports...
2022-11-23T02:59:56.1961836Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024802.xml
2022-11-23T02:59:56.1962213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1962391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1962818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1963011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1963031Z 
2022-11-23T02:59:56.1963122Z Running tests...
2022-11-23T02:59:56.1963386Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1963697Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1963932Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1964179Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1964400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86763
2022-11-23T02:59:56.1964618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86764
2022-11-23T02:59:56.1964989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1965145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1965519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1965709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1966075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1966247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1966618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1966853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1967081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1967290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1967546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmyxivz3t
2022-11-23T02:59:56.1967817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmyxivz3t/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1968071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq50l48tw
2022-11-23T02:59:56.1968334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq50l48tw/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1968566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1968798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1969029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1969264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1970173Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:59:56.1970270Z   warnings.warn(
2022-11-23T02:59:56.1971168Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:59:56.1971284Z   warnings.warn(
2022-11-23T02:59:56.1971558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1971784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1972012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1972244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1972476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1972700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1972901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1973129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1973231Z ok (6.100s)
2022-11-23T02:59:56.1973251Z 
2022-11-23T02:59:56.1973520Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1973633Z Ran 1 test in 6.100s
2022-11-23T02:59:56.1973652Z 
2022-11-23T02:59:56.1973746Z OK
2022-11-23T02:59:56.1973764Z 
2022-11-23T02:59:56.1973888Z Generating XML reports...
2022-11-23T02:59:56.1974350Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024811.xml
2022-11-23T02:59:56.1974722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1974879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1975256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1975491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1975511Z 
2022-11-23T02:59:56.1975621Z Running tests...
2022-11-23T02:59:56.1975885Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1976198Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1976461Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1976813Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1977013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86883
2022-11-23T02:59:56.1977228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86884
2022-11-23T02:59:56.1977597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1977776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1978393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1978582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1978949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1979123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1979493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1979693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1979921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1980150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1980404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2ewgn78
2022-11-23T02:59:56.1980740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2ewgn78/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1980998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9mwvinf
2022-11-23T02:59:56.1981264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9mwvinf/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1981496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1981704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1981936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1982162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1982269Z ok (6.046s)
2022-11-23T02:59:56.1982288Z 
2022-11-23T02:59:56.1982554Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1982667Z Ran 1 test in 6.046s
2022-11-23T02:59:56.1982690Z 
2022-11-23T02:59:56.1982785Z OK
2022-11-23T02:59:56.1982803Z 
2022-11-23T02:59:56.1982927Z Generating XML reports...
2022-11-23T02:59:56.1983367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024819.xml
2022-11-23T02:59:56.1983737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1983915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1984291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1984543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1984562Z 
2022-11-23T02:59:56.1984672Z Running tests...
2022-11-23T02:59:56.1984937Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1985251Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1985513Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1985845Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1986065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87003
2022-11-23T02:59:56.1986279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87004
2022-11-23T02:59:56.1986648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1986824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1987206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1987395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1987760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1987914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1988286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1988468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1988696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1988914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1989175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkix00sf
2022-11-23T02:59:56.1989443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkix00sf/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1989742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpju6cdr26
2022-11-23T02:59:56.1990014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpju6cdr26/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1990228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1990463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1990695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1990924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1991027Z ok (6.035s)
2022-11-23T02:59:56.1991050Z 
2022-11-23T02:59:56.1991317Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1991430Z Ran 1 test in 6.035s
2022-11-23T02:59:56.1991449Z 
2022-11-23T02:59:56.1991543Z OK
2022-11-23T02:59:56.1991562Z 
2022-11-23T02:59:56.1991669Z Generating XML reports...
2022-11-23T02:59:56.1992129Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024828.xml
2022-11-23T02:59:56.1992494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1992670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1993044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1993235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1993254Z 
2022-11-23T02:59:56.1993408Z Running tests...
2022-11-23T02:59:56.1993672Z ----------------------------------------------------------------------
2022-11-23T02:59:56.1993981Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.1994206Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.1994577Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.1994794Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87123
2022-11-23T02:59:56.1995009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87124
2022-11-23T02:59:56.1995377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1995554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1995931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1996125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1996469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.1996643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.1997014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.1997203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.1997430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.1997658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.1997915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdn4atake
2022-11-23T02:59:56.1998185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdn4atake/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1998435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50g4tql7
2022-11-23T02:59:56.1998739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50g4tql7/_remote_module_non_scriptable.py
2022-11-23T02:59:56.1998979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1999210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.1999983Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:59:56.2000771Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:59:56.2001007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2001238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2001340Z ok (6.134s)
2022-11-23T02:59:56.2001419Z 
2022-11-23T02:59:56.2001691Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2001806Z Ran 1 test in 6.135s
2022-11-23T02:59:56.2001825Z 
2022-11-23T02:59:56.2001918Z OK
2022-11-23T02:59:56.2001936Z 
2022-11-23T02:59:56.2002060Z Generating XML reports...
2022-11-23T02:59:56.2002506Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024836.xml
2022-11-23T02:59:56.2002873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2003051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2003430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2003621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2003640Z 
2022-11-23T02:59:56.2003750Z Running tests...
2022-11-23T02:59:56.2004018Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2004329Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2004551Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2004926Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2005146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87243
2022-11-23T02:59:56.2005363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87244
2022-11-23T02:59:56.2005733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2005907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2006284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2006478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2006842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2007040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2007418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2007603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2007831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2008057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2008313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxio0s_q
2022-11-23T02:59:56.2008588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxio0s_q/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2008841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6p9r34vr
2022-11-23T02:59:56.2009089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6p9r34vr/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2009194Z ok (6.048s)
2022-11-23T02:59:56.2009213Z 
2022-11-23T02:59:56.2009480Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2009595Z Ran 1 test in 6.048s
2022-11-23T02:59:56.2009614Z 
2022-11-23T02:59:56.2009707Z OK
2022-11-23T02:59:56.2009726Z 
2022-11-23T02:59:56.2009850Z Generating XML reports...
2022-11-23T02:59:56.2010309Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024844.xml
2022-11-23T02:59:56.2010676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2010899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2011255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2011449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2011468Z 
2022-11-23T02:59:56.2011577Z Running tests...
2022-11-23T02:59:56.2011839Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2012151Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2012388Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2012658Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2012874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87363
2022-11-23T02:59:56.2013074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87364
2022-11-23T02:59:56.2013445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2013621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2014000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2014190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2014550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2014721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2015093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2015282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2015492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2015718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2016026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbr0ujoq8
2022-11-23T02:59:56.2016299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbr0ujoq8/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2016551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4d1w6t8
2022-11-23T02:59:56.2016815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4d1w6t8/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2017048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2017283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2017499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2017728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2017833Z ok (6.033s)
2022-11-23T02:59:56.2017853Z 
2022-11-23T02:59:56.2018346Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2018461Z Ran 1 test in 6.033s
2022-11-23T02:59:56.2018481Z 
2022-11-23T02:59:56.2018575Z OK
2022-11-23T02:59:56.2018594Z 
2022-11-23T02:59:56.2018720Z Generating XML reports...
2022-11-23T02:59:56.2019180Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024853.xml
2022-11-23T02:59:56.2019549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2019707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2020175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2020364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2020383Z 
2022-11-23T02:59:56.2020493Z Running tests...
2022-11-23T02:59:56.2020755Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2021068Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2021323Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2021590Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2021789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87483
2022-11-23T02:59:56.2022006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87484
2022-11-23T02:59:56.2022378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2022553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2022932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2023122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2023483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2023653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2024026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2024194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2024421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2024650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2024905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40wc45tk
2022-11-23T02:59:56.2025229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40wc45tk/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2025487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lq_s6g2
2022-11-23T02:59:56.2025750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lq_s6g2/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2026528Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:59:56.2027304Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:59:56.2028216Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:59:56.2028376Z   warnings.warn(
2022-11-23T02:59:56.2029285Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:59:56.2029398Z   warnings.warn(
2022-11-23T02:59:56.2029633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2029849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2030083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2030318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2030421Z ok (6.038s)
2022-11-23T02:59:56.2030440Z 
2022-11-23T02:59:56.2030709Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2030826Z Ran 1 test in 6.039s
2022-11-23T02:59:56.2030845Z 
2022-11-23T02:59:56.2030939Z OK
2022-11-23T02:59:56.2030958Z 
2022-11-23T02:59:56.2031086Z Generating XML reports...
2022-11-23T02:59:56.2031526Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024901.xml
2022-11-23T02:59:56.2031894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2032069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2032447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2032641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2032661Z 
2022-11-23T02:59:56.2032771Z Running tests...
2022-11-23T02:59:56.2033035Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2033391Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2033650Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2033900Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2034118Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87603
2022-11-23T02:59:56.2034337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87604
2022-11-23T02:59:56.2034708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2034886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2035261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2035454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2035816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2035967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2036334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2036523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2036753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2036981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2037283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplws7sbhj
2022-11-23T02:59:56.2037552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplws7sbhj/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2037807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nm0z72v
2022-11-23T02:59:56.2038071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nm0z72v/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2038959Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:59:56.2039074Z   warnings.warn(
2022-11-23T02:59:56.2039987Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:59:56.2040102Z   warnings.warn(
2022-11-23T02:59:56.2040344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2040575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2040803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2041036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2041140Z ok (6.041s)
2022-11-23T02:59:56.2041159Z 
2022-11-23T02:59:56.2041431Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2041525Z Ran 1 test in 6.041s
2022-11-23T02:59:56.2041544Z 
2022-11-23T02:59:56.2041638Z OK
2022-11-23T02:59:56.2041657Z 
2022-11-23T02:59:56.2041781Z Generating XML reports...
2022-11-23T02:59:56.2042289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024910.xml
2022-11-23T02:59:56.2042664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2042839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2043215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2043407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2043427Z 
2022-11-23T02:59:56.2043535Z Running tests...
2022-11-23T02:59:56.2043782Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2044091Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2044350Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2044588Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2044806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87723
2022-11-23T02:59:56.2045020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87724
2022-11-23T02:59:56.2045387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2045562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2045918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2046156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2046514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2046689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2047056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2047243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2047474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2047696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2047931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8aeup6cv
2022-11-23T02:59:56.2048199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8aeup6cv/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2048454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjpami_md
2022-11-23T02:59:56.2048722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjpami_md/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2048959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2049191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2049422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2049653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2049755Z ok (6.013s)
2022-11-23T02:59:56.2049776Z 
2022-11-23T02:59:56.2050023Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2050133Z Ran 1 test in 6.013s
2022-11-23T02:59:56.2050156Z 
2022-11-23T02:59:56.2050249Z OK
2022-11-23T02:59:56.2050268Z 
2022-11-23T02:59:56.2050394Z Generating XML reports...
2022-11-23T02:59:56.2050851Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024918.xml
2022-11-23T02:59:56.2051264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2051446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2051827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2051996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2052035Z 
2022-11-23T02:59:56.2052126Z Running tests...
2022-11-23T02:59:56.2052388Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2052696Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2052958Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2053193Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2053413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87843
2022-11-23T02:59:56.2053630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87844
2022-11-23T02:59:56.2054000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2054156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2054533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2054722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2055133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2055305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2055678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2055870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2056098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2056305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2056560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwayqemhl
2022-11-23T02:59:56.2056827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwayqemhl/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2057081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpplkj6pyh
2022-11-23T02:59:56.2057354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpplkj6pyh/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2057589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2057824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2058286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2058522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2058729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2058954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2059180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2059412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2059515Z ok (6.013s)
2022-11-23T02:59:56.2059535Z 
2022-11-23T02:59:56.2059806Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2059988Z Ran 1 test in 6.013s
2022-11-23T02:59:56.2060009Z 
2022-11-23T02:59:56.2060106Z OK
2022-11-23T02:59:56.2060125Z 
2022-11-23T02:59:56.2060231Z Generating XML reports...
2022-11-23T02:59:56.2060696Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024926.xml
2022-11-23T02:59:56.2061062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2061236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2061614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2061807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2061826Z 
2022-11-23T02:59:56.2061936Z Running tests...
2022-11-23T02:59:56.2062197Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2062510Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2062779Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2062998Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87963
2022-11-23T02:59:56.2063213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87964
2022-11-23T02:59:56.2063583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2063757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2064202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2064392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2064755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2064907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2065277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2065462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2065688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2065906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2066165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlj8o2sw
2022-11-23T02:59:56.2066439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlj8o2sw/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2066692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8ub8n60
2022-11-23T02:59:56.2066960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8ub8n60/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2067047Z ok (6.652s)
2022-11-23T02:59:56.2067067Z 
2022-11-23T02:59:56.2067336Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2067452Z Ran 1 test in 6.652s
2022-11-23T02:59:56.2067472Z 
2022-11-23T02:59:56.2067565Z OK
2022-11-23T02:59:56.2067583Z 
2022-11-23T02:59:56.2067707Z Generating XML reports...
2022-11-23T02:59:56.2068163Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024935.xml
2022-11-23T02:59:56.2068531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2068708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2069114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2069311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2069331Z 
2022-11-23T02:59:56.2069440Z Running tests...
2022-11-23T02:59:56.2069703Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2070019Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2070320Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2070540Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88084
2022-11-23T02:59:56.2070762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88085
2022-11-23T02:59:56.2071134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2071292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2071665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2071855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2072219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2072390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2072760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2072949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2073230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2073437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2073697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefe3cn7y
2022-11-23T02:59:56.2073964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefe3cn7y/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2074216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqqa48x_
2022-11-23T02:59:56.2074485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqqa48x_/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2074589Z ok (6.545s)
2022-11-23T02:59:56.2074608Z 
2022-11-23T02:59:56.2074875Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2074989Z Ran 1 test in 6.545s
2022-11-23T02:59:56.2075012Z 
2022-11-23T02:59:56.2075086Z OK
2022-11-23T02:59:56.2075123Z 
2022-11-23T02:59:56.2075227Z Generating XML reports...
2022-11-23T02:59:56.2075683Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024944.xml
2022-11-23T02:59:56.2076056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2076231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2076612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2076802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2076821Z 
2022-11-23T02:59:56.2076931Z Running tests...
2022-11-23T02:59:56.2077197Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2077490Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2077798Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2078013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88205
2022-11-23T02:59:56.2078271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88206
2022-11-23T02:59:56.2078646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2078820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2079194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2079384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2079788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2079947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2080315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2080505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2080734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2080954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2081214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yds8fac
2022-11-23T02:59:56.2081482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yds8fac/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2081732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8au3mois
2022-11-23T02:59:56.2082046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8au3mois/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2082153Z ok (6.534s)
2022-11-23T02:59:56.2082173Z 
2022-11-23T02:59:56.2082439Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2082552Z Ran 1 test in 6.534s
2022-11-23T02:59:56.2082574Z 
2022-11-23T02:59:56.2082667Z OK
2022-11-23T02:59:56.2082686Z 
2022-11-23T02:59:56.2082813Z Generating XML reports...
2022-11-23T02:59:56.2083277Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024953.xml
2022-11-23T02:59:56.2083642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2083816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2084176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2084371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2084390Z 
2022-11-23T02:59:56.2084499Z Running tests...
2022-11-23T02:59:56.2084757Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2085074Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2085311Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2085595Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2085811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88326
2022-11-23T02:59:56.2086008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88327
2022-11-23T02:59:56.2086378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2086553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2086928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2087162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2087528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2087700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2088076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2088263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2088470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2088696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2088954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4t5erm22
2022-11-23T02:59:56.2089219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4t5erm22/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2089475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhzwctwi
2022-11-23T02:59:56.2089744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhzwctwi/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2089848Z ok (6.638s)
2022-11-23T02:59:56.2089867Z 
2022-11-23T02:59:56.2090135Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2090229Z Ran 1 test in 6.638s
2022-11-23T02:59:56.2090248Z 
2022-11-23T02:59:56.2090342Z OK
2022-11-23T02:59:56.2090360Z 
2022-11-23T02:59:56.2090483Z Generating XML reports...
2022-11-23T02:59:56.2090941Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025001.xml
2022-11-23T02:59:56.2091359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2091535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2091911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2092101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2092121Z 
2022-11-23T02:59:56.2092230Z Running tests...
2022-11-23T02:59:56.2092473Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2092784Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2093014Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2093306Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2093529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88447
2022-11-23T02:59:56.2093744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88448
2022-11-23T02:59:56.2094120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2094297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2094656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2094844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2095209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2095382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2095758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2095944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2096214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2096449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2096704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajjopqhg
2022-11-23T02:59:56.2096955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajjopqhg/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2097209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljcpuytq
2022-11-23T02:59:56.2097474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljcpuytq/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2097582Z ok (6.611s)
2022-11-23T02:59:56.2097602Z 
2022-11-23T02:59:56.2097868Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2098190Z Ran 1 test in 6.611s
2022-11-23T02:59:56.2098211Z 
2022-11-23T02:59:56.2098303Z OK
2022-11-23T02:59:56.2098321Z 
2022-11-23T02:59:56.2098450Z Generating XML reports...
2022-11-23T02:59:56.2098901Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025010.xml
2022-11-23T02:59:56.2099274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2099449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2099823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2100012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2100031Z 
2022-11-23T02:59:56.2100213Z Running tests...
2022-11-23T02:59:56.2100477Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2100790Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2101077Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2101276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88568
2022-11-23T02:59:56.2101490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88569
2022-11-23T02:59:56.2101861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2102039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2102414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2102607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2102970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2103144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2103497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2103687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2103912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2104132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2104282Z skip: Need at least 4 CUDA devices (3.902s)
2022-11-23T02:59:56.2104301Z 
2022-11-23T02:59:56.2104564Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2104680Z Ran 1 test in 3.902s
2022-11-23T02:59:56.2104699Z 
2022-11-23T02:59:56.2104809Z OK (skipped=1)
2022-11-23T02:59:56.2104828Z 
2022-11-23T02:59:56.2104951Z Generating XML reports...
2022-11-23T02:59:56.2105449Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025019.xml
2022-11-23T02:59:56.2105824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2105995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2106373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2106564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2106583Z 
2022-11-23T02:59:56.2106693Z Running tests...
2022-11-23T02:59:56.2106955Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2107270Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2107454Z   test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2107712Z Tests that DDP with ``device_ids`` specified can run a forward and ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2107930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88671
2022-11-23T02:59:56.2108146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88672
2022-11-23T02:59:56.2108516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2108693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2109067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2109258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2109673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2109827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2110199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2110387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2110614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2110860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2111085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2111325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2111729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2112107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2112366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3uih68b0
2022-11-23T02:59:56.2112636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3uih68b0/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2112889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdm_y04o
2022-11-23T02:59:56.2113154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdm_y04o/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2114097Z /opt/conda/lib/python3.10/site-packages/torch/distributed/_shard/replicated_tensor.py:113: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cudnn/RNN.cpp:982.)
2022-11-23T02:59:56.2114222Z   rs = func(*args, **kwargs)
2022-11-23T02:59:56.2115206Z /opt/conda/lib/python3.10/site-packages/torch/distributed/_shard/replicated_tensor.py:113: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cudnn/RNN.cpp:982.)
2022-11-23T02:59:56.2115329Z   rs = func(*args, **kwargs)
2022-11-23T02:59:56.2115432Z ok (7.409s)
2022-11-23T02:59:56.2115453Z 
2022-11-23T02:59:56.2115718Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2115812Z Ran 1 test in 7.409s
2022-11-23T02:59:56.2115835Z 
2022-11-23T02:59:56.2115929Z OK
2022-11-23T02:59:56.2115949Z 
2022-11-23T02:59:56.2116075Z Generating XML reports...
2022-11-23T02:59:56.2116534Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025025.xml
2022-11-23T02:59:56.2116906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2117080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2117460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2117650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2117668Z 
2022-11-23T02:59:56.2117779Z Running tests...
2022-11-23T02:59:56.2118020Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2118329Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2118648Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2118868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88792
2022-11-23T02:59:56.2119086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88793
2022-11-23T02:59:56.2119454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2119630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2120003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2120175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2120537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2120715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2121084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2121274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2121500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2121726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2121976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptx_9sko2
2022-11-23T02:59:56.2122241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptx_9sko2/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2122474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbggft39
2022-11-23T02:59:56.2122741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbggft39/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2122979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2123214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2123491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2123724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2123948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2124175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2124382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2124612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2124718Z ok (6.943s)
2022-11-23T02:59:56.2124737Z 
2022-11-23T02:59:56.2125006Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2125121Z Ran 1 test in 6.943s
2022-11-23T02:59:56.2125139Z 
2022-11-23T02:59:56.2125233Z OK
2022-11-23T02:59:56.2125252Z 
2022-11-23T02:59:56.2125380Z Generating XML reports...
2022-11-23T02:59:56.2125842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025035.xml
2022-11-23T02:59:56.2126195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2126371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2126746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2126936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2126999Z 
2022-11-23T02:59:56.2127110Z Running tests...
2022-11-23T02:59:56.2127373Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2127684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2127966Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2128183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88913
2022-11-23T02:59:56.2128378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88914
2022-11-23T02:59:56.2128748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2128922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2129298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2129490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2129850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2130025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2130397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2130563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2130791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2131324Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-11-23T02:59:56.2131596Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-11-23T02:59:56.2131854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetfm_7f7
2022-11-23T02:59:56.2132122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetfm_7f7/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2132392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2132926Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-11-23T02:59:56.2133194Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-11-23T02:59:56.2133448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbse48wfw
2022-11-23T02:59:56.2133695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbse48wfw/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2133799Z ok (3.931s)
2022-11-23T02:59:56.2133823Z 
2022-11-23T02:59:56.2134084Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2134197Z Ran 1 test in 3.931s
2022-11-23T02:59:56.2134215Z 
2022-11-23T02:59:56.2134308Z OK
2022-11-23T02:59:56.2134327Z 
2022-11-23T02:59:56.2134452Z Generating XML reports...
2022-11-23T02:59:56.2134914Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025044.xml
2022-11-23T02:59:56.2135279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2135434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2135811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2136000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2136020Z 
2022-11-23T02:59:56.2136130Z Running tests...
2022-11-23T02:59:56.2136439Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2136752Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2137032Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2137248Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89020
2022-11-23T02:59:56.2137463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89021
2022-11-23T02:59:56.2137812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2138203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2138588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2138779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2139147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2139319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2139700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2139889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2140097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2140324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2140576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26f1_1s1
2022-11-23T02:59:56.2140838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26f1_1s1/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2141094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ipqf755
2022-11-23T02:59:56.2141359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ipqf755/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2141463Z ok (6.640s)
2022-11-23T02:59:56.2141549Z 
2022-11-23T02:59:56.2141825Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2141938Z Ran 1 test in 6.641s
2022-11-23T02:59:56.2141958Z 
2022-11-23T02:59:56.2142032Z OK
2022-11-23T02:59:56.2142051Z 
2022-11-23T02:59:56.2142176Z Generating XML reports...
2022-11-23T02:59:56.2142639Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025051.xml
2022-11-23T02:59:56.2143008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2143182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2143559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2143747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2143767Z 
2022-11-23T02:59:56.2143880Z Running tests...
2022-11-23T02:59:56.2144122Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2144431Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2144713Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2144928Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89141
2022-11-23T02:59:56.2145143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89142
2022-11-23T02:59:56.2145506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2145753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2146130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2146321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2146660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2146833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2147202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2147387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2147611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2147836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2148095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalffurcm
2022-11-23T02:59:56.2148368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalffurcm/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2148600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56gwp88t
2022-11-23T02:59:56.2148865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56gwp88t/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2148968Z ok (6.511s)
2022-11-23T02:59:56.2148987Z 
2022-11-23T02:59:56.2149251Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2149363Z Ran 1 test in 6.512s
2022-11-23T02:59:56.2149382Z 
2022-11-23T02:59:56.2149472Z OK
2022-11-23T02:59:56.2149491Z 
2022-11-23T02:59:56.2149612Z Generating XML reports...
2022-11-23T02:59:56.2150069Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025100.xml
2022-11-23T02:59:56.2150438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2150637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2151019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2151206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2151226Z 
2022-11-23T02:59:56.2151333Z Running tests...
2022-11-23T02:59:56.2151594Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2151903Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2152165Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2152386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89262
2022-11-23T02:59:56.2152583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89263
2022-11-23T02:59:56.2152952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2153132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2153506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2153695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2154056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2154227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2154598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2154834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2155043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2155267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2155522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2l08xso
2022-11-23T02:59:56.2155790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2l08xso/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2156040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1gtvn73s
2022-11-23T02:59:56.2156303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1gtvn73s/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2156534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2156763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2156973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2157204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2157307Z ok (7.230s)
2022-11-23T02:59:56.2157326Z 
2022-11-23T02:59:56.2157591Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2157703Z Ran 1 test in 7.230s
2022-11-23T02:59:56.2157722Z 
2022-11-23T02:59:56.2157814Z OK
2022-11-23T02:59:56.2157832Z 
2022-11-23T02:59:56.2157957Z Generating XML reports...
2022-11-23T02:59:56.2158415Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025109.xml
2022-11-23T02:59:56.2158783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2158942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2159316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2159558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2159579Z 
2022-11-23T02:59:56.2159690Z Running tests...
2022-11-23T02:59:56.2159954Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2160261Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2160563Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2161325Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82632 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.584s)
2022-11-23T02:59:56.2161350Z 
2022-11-23T02:59:56.2161609Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2161703Z Ran 1 test in 1.584s
2022-11-23T02:59:56.2161741Z 
2022-11-23T02:59:56.2161835Z OK (skipped=1)
2022-11-23T02:59:56.2161854Z 
2022-11-23T02:59:56.2161977Z Generating XML reports...
2022-11-23T02:59:56.2162431Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025118.xml
2022-11-23T02:59:56.2162797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2162969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2163345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2163535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2163603Z 
2022-11-23T02:59:56.2163714Z Running tests...
2022-11-23T02:59:56.2163959Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2164272Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2164565Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2165304Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/83301 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-11-23T02:59:56.2165324Z 
2022-11-23T02:59:56.2165584Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2165699Z Ran 1 test in 1.640s
2022-11-23T02:59:56.2165721Z 
2022-11-23T02:59:56.2165828Z OK (skipped=1)
2022-11-23T02:59:56.2165847Z 
2022-11-23T02:59:56.2165968Z Generating XML reports...
2022-11-23T02:59:56.2166421Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025122.xml
2022-11-23T02:59:56.2166790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2166947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2167322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2167510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2167529Z 
2022-11-23T02:59:56.2167639Z Running tests...
2022-11-23T02:59:56.2167899Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2168209Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2168504Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2169291Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82385 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.625s)
2022-11-23T02:59:56.2169314Z 
2022-11-23T02:59:56.2169575Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2169668Z Ran 1 test in 1.625s
2022-11-23T02:59:56.2169706Z 
2022-11-23T02:59:56.2169796Z OK (skipped=1)
2022-11-23T02:59:56.2169815Z 
2022-11-23T02:59:56.2169937Z Generating XML reports...
2022-11-23T02:59:56.2170394Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025126.xml
2022-11-23T02:59:56.2170767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2170942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2171323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2171515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2171534Z 
2022-11-23T02:59:56.2171645Z Running tests...
2022-11-23T02:59:56.2171886Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2172198Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2172513Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2173252Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82979 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-11-23T02:59:56.2173320Z 
2022-11-23T02:59:56.2173585Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2173700Z Ran 1 test in 1.612s
2022-11-23T02:59:56.2173719Z 
2022-11-23T02:59:56.2173827Z OK (skipped=1)
2022-11-23T02:59:56.2173846Z 
2022-11-23T02:59:56.2173969Z Generating XML reports...
2022-11-23T02:59:56.2174424Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025130.xml
2022-11-23T02:59:56.2174794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2174951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2175331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2175520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2175539Z 
2022-11-23T02:59:56.2175652Z Running tests...
2022-11-23T02:59:56.2175913Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2176223Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2176534Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2177269Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82400 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2022-11-23T02:59:56.2177292Z 
2022-11-23T02:59:56.2177553Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2177647Z Ran 1 test in 1.630s
2022-11-23T02:59:56.2177684Z 
2022-11-23T02:59:56.2177774Z OK (skipped=1)
2022-11-23T02:59:56.2177793Z 
2022-11-23T02:59:56.2178217Z Generating XML reports...
2022-11-23T02:59:56.2178696Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025134.xml
2022-11-23T02:59:56.2179061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2179236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2179611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2179841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2179865Z 
2022-11-23T02:59:56.2179976Z Running tests...
2022-11-23T02:59:56.2180222Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2180531Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2180844Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2181580Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82500 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2022-11-23T02:59:56.2181600Z 
2022-11-23T02:59:56.2181859Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2181976Z Ran 1 test in 1.627s
2022-11-23T02:59:56.2181996Z 
2022-11-23T02:59:56.2182169Z OK (skipped=1)
2022-11-23T02:59:56.2182189Z 
2022-11-23T02:59:56.2182314Z Generating XML reports...
2022-11-23T02:59:56.2182767Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025138.xml
2022-11-23T02:59:56.2183137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2183292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2183671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2183857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2183877Z 
2022-11-23T02:59:56.2183986Z Running tests...
2022-11-23T02:59:56.2184248Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2184555Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2184802Z   test_fp16 (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2185021Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89598
2022-11-23T02:59:56.2185221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89599
2022-11-23T02:59:56.2185589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2185757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2186134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2186321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2186688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2186863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2187231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2187415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2187667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2187904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2188159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4gkqaoh
2022-11-23T02:59:56.2188427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4gkqaoh/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2188679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwnmx3ee
2022-11-23T02:59:56.2188946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwnmx3ee/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2189054Z ok (7.133s)
2022-11-23T02:59:56.2189074Z 
2022-11-23T02:59:56.2189337Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2189431Z Ran 1 test in 7.133s
2022-11-23T02:59:56.2189471Z 
2022-11-23T02:59:56.2189547Z OK
2022-11-23T02:59:56.2189568Z 
2022-11-23T02:59:56.2189692Z Generating XML reports...
2022-11-23T02:59:56.2190146Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025142.xml
2022-11-23T02:59:56.2190513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2190689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2191065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2191254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2191317Z 
2022-11-23T02:59:56.2191431Z Running tests...
2022-11-23T02:59:56.2191677Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2191984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2192269Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2192485Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89719
2022-11-23T02:59:56.2192699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89720
2022-11-23T02:59:56.2193065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2193241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2193618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2193792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2194156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2194330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2194704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2194891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2195119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2195339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2195931Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2196511Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2196771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozj0vqrc
2022-11-23T02:59:56.2197040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozj0vqrc/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2197273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponzidjp1
2022-11-23T02:59:56.2197544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponzidjp1/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2197652Z ok (6.612s)
2022-11-23T02:59:56.2197671Z 
2022-11-23T02:59:56.2197938Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2198051Z Ran 1 test in 6.612s
2022-11-23T02:59:56.2198073Z 
2022-11-23T02:59:56.2198169Z OK
2022-11-23T02:59:56.2198187Z 
2022-11-23T02:59:56.2198311Z Generating XML reports...
2022-11-23T02:59:56.2198767Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025151.xml
2022-11-23T02:59:56.2199116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2199291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2199669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2199903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2199923Z 
2022-11-23T02:59:56.2200033Z Running tests...
2022-11-23T02:59:56.2200295Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2200610Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2200888Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2201105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89840
2022-11-23T02:59:56.2201302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89841
2022-11-23T02:59:56.2201668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2201844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2202217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2202410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2202776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2202950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2203318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2203486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2203713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2204253Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2204482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2205059Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2205317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yj30qz8
2022-11-23T02:59:56.2205588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yj30qz8/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2205838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4x759nhe
2022-11-23T02:59:56.2206112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4x759nhe/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2206216Z ok (6.663s)
2022-11-23T02:59:56.2206236Z 
2022-11-23T02:59:56.2206504Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2206600Z Ran 1 test in 6.664s
2022-11-23T02:59:56.2206620Z 
2022-11-23T02:59:56.2206714Z OK
2022-11-23T02:59:56.2206733Z 
2022-11-23T02:59:56.2206857Z Generating XML reports...
2022-11-23T02:59:56.2207315Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025200.xml
2022-11-23T02:59:56.2207688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2207863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2208238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2208492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2208511Z 
2022-11-23T02:59:56.2208622Z Running tests...
2022-11-23T02:59:56.2208866Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2209180Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2209441Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2209659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89961
2022-11-23T02:59:56.2209873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89962
2022-11-23T02:59:56.2210236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2210410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2210781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2210955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2211319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2211493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2211865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2212052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2212278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2212505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2212757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73k1yduu
2022-11-23T02:59:56.2213007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73k1yduu/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2213260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuwb2r9r
2022-11-23T02:59:56.2213573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuwb2r9r/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2213683Z ok (7.047s)
2022-11-23T02:59:56.2213702Z 
2022-11-23T02:59:56.2213966Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2214080Z Ran 1 test in 7.047s
2022-11-23T02:59:56.2214099Z 
2022-11-23T02:59:56.2214192Z OK
2022-11-23T02:59:56.2214211Z 
2022-11-23T02:59:56.2214335Z Generating XML reports...
2022-11-23T02:59:56.2214792Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025209.xml
2022-11-23T02:59:56.2215141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2215318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2215698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2215889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2215908Z 
2022-11-23T02:59:56.2216018Z Running tests...
2022-11-23T02:59:56.2216280Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2216591Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2216900Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2217099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90082
2022-11-23T02:59:56.2217316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90083
2022-11-23T02:59:56.2217734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2218125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2218515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2218706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2219068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2219240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2219617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2219790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2220021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2220248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2220504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmsj7bz6
2022-11-23T02:59:56.2220773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmsj7bz6/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2221027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfy967pz
2022-11-23T02:59:56.2221292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfy967pz/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2221527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2221760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2221977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2222204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2222435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2222731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2222963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2223187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2223408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2223638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2223842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2224072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2224294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2224518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2224745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2224970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2225194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2225416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2225616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2225841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2226120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2226345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2226570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2226791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2227017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2227239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2227463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2227667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2227888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2228114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2228338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2228565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2228787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2229009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2229232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2229438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2229542Z ok (8.622s)
2022-11-23T02:59:56.2229561Z 
2022-11-23T02:59:56.2229831Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2229951Z Ran 1 test in 8.622s
2022-11-23T02:59:56.2229970Z 
2022-11-23T02:59:56.2230063Z OK
2022-11-23T02:59:56.2230081Z 
2022-11-23T02:59:56.2230205Z Generating XML reports...
2022-11-23T02:59:56.2230710Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025218.xml
2022-11-23T02:59:56.2231091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2231246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2231622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2231811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2231831Z 
2022-11-23T02:59:56.2231941Z Running tests...
2022-11-23T02:59:56.2232203Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2232520Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2232797Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2233020Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90203
2022-11-23T02:59:56.2233236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90204
2022-11-23T02:59:56.2233584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2233759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2234138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2234326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2234691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2234910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2235283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2235469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2235677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2235901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2236050Z skip: Need at least 4 CUDA devices (3.937s)
2022-11-23T02:59:56.2236069Z 
2022-11-23T02:59:56.2236332Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2236446Z Ran 1 test in 3.937s
2022-11-23T02:59:56.2236465Z 
2022-11-23T02:59:56.2236574Z OK (skipped=1)
2022-11-23T02:59:56.2236597Z 
2022-11-23T02:59:56.2236724Z Generating XML reports...
2022-11-23T02:59:56.2237183Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025229.xml
2022-11-23T02:59:56.2237554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2237709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2238085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2238276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2238295Z 
2022-11-23T02:59:56.2238405Z Running tests...
2022-11-23T02:59:56.2238667Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2238979Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2239256Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2239472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90306
2022-11-23T02:59:56.2239713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90307
2022-11-23T02:59:56.2240089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2240262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2240626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2240799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2241177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2241369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2241745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2241935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2242144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2242685Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2243225Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2243803Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2244341Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2244874Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2245416Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2245643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2246171Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2246742Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2247268Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2247799Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2248333Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2248862Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2248969Z ok (3.944s)
2022-11-23T02:59:56.2248990Z 
2022-11-23T02:59:56.2249255Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2249408Z Ran 1 test in 3.945s
2022-11-23T02:59:56.2249428Z 
2022-11-23T02:59:56.2249521Z OK
2022-11-23T02:59:56.2249539Z 
2022-11-23T02:59:56.2249664Z Generating XML reports...
2022-11-23T02:59:56.2250129Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025236.xml
2022-11-23T02:59:56.2250498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2250654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2251030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2251219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2251238Z 
2022-11-23T02:59:56.2251348Z Running tests...
2022-11-23T02:59:56.2251610Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2251927Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2252221Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2252443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90409
2022-11-23T02:59:56.2252639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90410
2022-11-23T02:59:56.2253008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2253178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2253554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2253742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2254103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2254278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2254694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2254883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2255091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2255308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2255566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6uyd9u9
2022-11-23T02:59:56.2255842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6uyd9u9/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2256094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44ejcpcm
2022-11-23T02:59:56.2256367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44ejcpcm/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2256471Z ok (7.152s)
2022-11-23T02:59:56.2256491Z 
2022-11-23T02:59:56.2256763Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2256858Z Ran 1 test in 7.152s
2022-11-23T02:59:56.2256896Z 
2022-11-23T02:59:56.2256970Z OK
2022-11-23T02:59:56.2256989Z 
2022-11-23T02:59:56.2257112Z Generating XML reports...
2022-11-23T02:59:56.2257568Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025242.xml
2022-11-23T02:59:56.2258139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2258320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2258701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2258967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2258986Z 
2022-11-23T02:59:56.2259096Z Running tests...
2022-11-23T02:59:56.2259343Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2259652Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2259958Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2260176Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90530
2022-11-23T02:59:56.2260393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90531
2022-11-23T02:59:56.2260764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2260939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2261320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2261490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2261857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2262031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2262408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2263519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2263747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2263973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2264233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuoyzal4k
2022-11-23T02:59:56.2264502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuoyzal4k/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2264800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_gv85r1f
2022-11-23T02:59:56.2265074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_gv85r1f/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2265177Z ok (7.039s)
2022-11-23T02:59:56.2265197Z 
2022-11-23T02:59:56.2265465Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2265580Z Ran 1 test in 7.039s
2022-11-23T02:59:56.2265599Z 
2022-11-23T02:59:56.2265690Z OK
2022-11-23T02:59:56.2265708Z 
2022-11-23T02:59:56.2265831Z Generating XML reports...
2022-11-23T02:59:56.2266288Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025252.xml
2022-11-23T02:59:56.2266660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2266815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2267190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2267375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2267395Z 
2022-11-23T02:59:56.2267501Z Running tests...
2022-11-23T02:59:56.2267760Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2268069Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2268372Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2268589Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90651
2022-11-23T02:59:56.2268898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90652
2022-11-23T02:59:56.2269269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2269447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2269822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2270008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2270368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2270540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2270910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2271099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2271306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2271532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2271789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphf4o4rz3
2022-11-23T02:59:56.2272056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphf4o4rz3/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2272305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9l5gd1u
2022-11-23T02:59:56.2272570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9l5gd1u/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2272802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2273033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2273121Z ok (7.051s)
2022-11-23T02:59:56.2273157Z 
2022-11-23T02:59:56.2273405Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2273516Z Ran 1 test in 7.052s
2022-11-23T02:59:56.2273536Z 
2022-11-23T02:59:56.2273676Z OK
2022-11-23T02:59:56.2273696Z 
2022-11-23T02:59:56.2273824Z Generating XML reports...
2022-11-23T02:59:56.2274282Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025301.xml
2022-11-23T02:59:56.2274652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2274828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2275204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2275373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2275396Z 
2022-11-23T02:59:56.2275504Z Running tests...
2022-11-23T02:59:56.2275762Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2276072Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2276384Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2276601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90772
2022-11-23T02:59:56.2276814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90773
2022-11-23T02:59:56.2277182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2277336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2277708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2277947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2278313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2278485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2278852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2279036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2279265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2279489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2279765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nnz2r3z
2022-11-23T02:59:56.2280037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nnz2r3z/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2280288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqq611v97
2022-11-23T02:59:56.2280552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqq611v97/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2280785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2281018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2281119Z ok (7.006s)
2022-11-23T02:59:56.2281139Z 
2022-11-23T02:59:56.2281405Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2281499Z Ran 1 test in 7.006s
2022-11-23T02:59:56.2281534Z 
2022-11-23T02:59:56.2281609Z OK
2022-11-23T02:59:56.2281627Z 
2022-11-23T02:59:56.2281748Z Generating XML reports...
2022-11-23T02:59:56.2282213Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025310.xml
2022-11-23T02:59:56.2282578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2282798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2283180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2283370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2283389Z 
2022-11-23T02:59:56.2283499Z Running tests...
2022-11-23T02:59:56.2283745Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2284050Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2284320Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2284536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90893
2022-11-23T02:59:56.2284748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90894
2022-11-23T02:59:56.2285116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2285288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2285660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2285830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2286193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2286362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2286726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2286961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2287194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2287420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2287570Z skip: Need at least 4 CUDA devices (3.943s)
2022-11-23T02:59:56.2287590Z 
2022-11-23T02:59:56.2287853Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2287948Z Ran 1 test in 3.944s
2022-11-23T02:59:56.2287966Z 
2022-11-23T02:59:56.2288074Z OK (skipped=1)
2022-11-23T02:59:56.2288093Z 
2022-11-23T02:59:56.2288217Z Generating XML reports...
2022-11-23T02:59:56.2288674Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025320.xml
2022-11-23T02:59:56.2289045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2289218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2289596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2289777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2289796Z 
2022-11-23T02:59:56.2289908Z Running tests...
2022-11-23T02:59:56.2290169Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2290478Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2290748Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2290945Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90996
2022-11-23T02:59:56.2291162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90997
2022-11-23T02:59:56.2291531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2291747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2292129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2292317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2292678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2292850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2293202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2293391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2293617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2293840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2293991Z skip: Need at least 8 CUDA devices (3.999s)
2022-11-23T02:59:56.2294011Z 
2022-11-23T02:59:56.2294274Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2294388Z Ran 1 test in 3.999s
2022-11-23T02:59:56.2294407Z 
2022-11-23T02:59:56.2294512Z OK (skipped=1)
2022-11-23T02:59:56.2294531Z 
2022-11-23T02:59:56.2294653Z Generating XML reports...
2022-11-23T02:59:56.2295093Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025326.xml
2022-11-23T02:59:56.2295457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2295682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2296054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2296245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2296265Z 
2022-11-23T02:59:56.2296373Z Running tests...
2022-11-23T02:59:56.2296631Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2296937Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2297212Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2297426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91099
2022-11-23T02:59:56.2297637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91100
2022-11-23T02:59:56.2298223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2298403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2298786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2298974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2299339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2299507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2299856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2300041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2300266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2300492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2300593Z ok (5.409s)
2022-11-23T02:59:56.2300612Z 
2022-11-23T02:59:56.2300950Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2301068Z Ran 1 test in 5.409s
2022-11-23T02:59:56.2301087Z 
2022-11-23T02:59:56.2301179Z OK
2022-11-23T02:59:56.2301197Z 
2022-11-23T02:59:56.2301302Z Generating XML reports...
2022-11-23T02:59:56.2301762Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025332.xml
2022-11-23T02:59:56.2302124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2302296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2302660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2302849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2302868Z 
2022-11-23T02:59:56.2302981Z Running tests...
2022-11-23T02:59:56.2303241Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2303550Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2303835Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2304050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91208
2022-11-23T02:59:56.2304264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91209
2022-11-23T02:59:56.2304631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2304868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2305236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2305424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2305787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2305940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2306304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2306488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2306712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2306936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2307084Z skip: Need at least 4 CUDA devices (3.919s)
2022-11-23T02:59:56.2307104Z 
2022-11-23T02:59:56.2307366Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2307476Z Ran 1 test in 3.920s
2022-11-23T02:59:56.2307495Z 
2022-11-23T02:59:56.2307605Z OK (skipped=1)
2022-11-23T02:59:56.2307624Z 
2022-11-23T02:59:56.2307729Z Generating XML reports...
2022-11-23T02:59:56.2308186Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025340.xml
2022-11-23T02:59:56.2308552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2308725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2309099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2309286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2309309Z 
2022-11-23T02:59:56.2309416Z Running tests...
2022-11-23T02:59:56.2309675Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2310010Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2310319Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2310536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91311
2022-11-23T02:59:56.2310749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91312
2022-11-23T02:59:56.2311115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2311285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2311658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2311847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2312205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2312359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2312727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2312911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2313138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2313360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2313615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxe5jj63
2022-11-23T02:59:56.2313933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxe5jj63/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2314181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtyxntnf
2022-11-23T02:59:56.2314449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtyxntnf/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2314664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2314891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2314994Z ok (7.042s)
2022-11-23T02:59:56.2315013Z 
2022-11-23T02:59:56.2315279Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2315391Z Ran 1 test in 7.042s
2022-11-23T02:59:56.2315410Z 
2022-11-23T02:59:56.2315501Z OK
2022-11-23T02:59:56.2315520Z 
2022-11-23T02:59:56.2315642Z Generating XML reports...
2022-11-23T02:59:56.2316102Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025346.xml
2022-11-23T02:59:56.2316450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2316624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2316997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2317185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2317204Z 
2022-11-23T02:59:56.2317313Z Running tests...
2022-11-23T02:59:56.2317570Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2317877Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2318183Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2318400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91432
2022-11-23T02:59:56.2318598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91433
2022-11-23T02:59:56.2319010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2319186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2319560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2319752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2320110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2320281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2320651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2320819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2321046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2321268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2321521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij4jfvj8
2022-11-23T02:59:56.2321785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij4jfvj8/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2322033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe05ho8zo
2022-11-23T02:59:56.2322293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe05ho8zo/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2322523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2322804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2322890Z ok (7.153s)
2022-11-23T02:59:56.2322909Z 
2022-11-23T02:59:56.2323183Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2323294Z Ran 1 test in 7.153s
2022-11-23T02:59:56.2323313Z 
2022-11-23T02:59:56.2323405Z OK
2022-11-23T02:59:56.2323424Z 
2022-11-23T02:59:56.2323544Z Generating XML reports...
2022-11-23T02:59:56.2323999Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025356.xml
2022-11-23T02:59:56.2324365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2324538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2324896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2325086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2325105Z 
2022-11-23T02:59:56.2325211Z Running tests...
2022-11-23T02:59:56.2325475Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2325781Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2326058Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2326272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91553
2022-11-23T02:59:56.2326487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91554
2022-11-23T02:59:56.2326851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2327006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2327382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2327568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2327977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2328152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2328525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2328715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2328941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2329149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2329256Z ok (22.184s)
2022-11-23T02:59:56.2329275Z 
2022-11-23T02:59:56.2329536Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2329648Z Ran 1 test in 22.184s
2022-11-23T02:59:56.2329667Z 
2022-11-23T02:59:56.2329760Z OK
2022-11-23T02:59:56.2329779Z 
2022-11-23T02:59:56.2329903Z Generating XML reports...
2022-11-23T02:59:56.2330359Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025405.xml
2022-11-23T02:59:56.2330721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2330875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2331249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2331432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2331511Z 
2022-11-23T02:59:56.2331625Z Running tests...
2022-11-23T02:59:56.2331885Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2332192Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2332375Z   test_no_grad (__main__.DistributedDataParallelTest)
2022-11-23T02:59:56.2332623Z Note: this test can be sped up by only running it on a CPU module ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2332838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91674
2022-11-23T02:59:56.2333034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91675
2022-11-23T02:59:56.2333400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2333569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2333942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2334133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2334494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2334663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2335034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2335201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2335425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2335650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2335904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhu30iko
2022-11-23T02:59:56.2336173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhu30iko/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2336423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6r912bc8
2022-11-23T02:59:56.2336730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6r912bc8/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2336837Z ok (7.050s)
2022-11-23T02:59:56.2336856Z 
2022-11-23T02:59:56.2337120Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2337214Z Ran 1 test in 7.050s
2022-11-23T02:59:56.2337233Z 
2022-11-23T02:59:56.2337324Z OK
2022-11-23T02:59:56.2337343Z 
2022-11-23T02:59:56.2337466Z Generating XML reports...
2022-11-23T02:59:56.2338121Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025430.xml
2022-11-23T02:59:56.2338499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2338676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2339048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2339238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2339258Z 
2022-11-23T02:59:56.2339348Z Running tests...
2022-11-23T02:59:56.2339606Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2339912Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2340187Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2340403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91791
2022-11-23T02:59:56.2340618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91792
2022-11-23T02:59:56.2341062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2341234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2341608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2341778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2342135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2342308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2342678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2342862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2343090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2343304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2343560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8jgnmyg
2022-11-23T02:59:56.2343807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8jgnmyg/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2344058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwass7cx
2022-11-23T02:59:56.2344320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwass7cx/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2344421Z ok (6.602s)
2022-11-23T02:59:56.2344440Z 
2022-11-23T02:59:56.2344705Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2344815Z Ran 1 test in 6.602s
2022-11-23T02:59:56.2344834Z 
2022-11-23T02:59:56.2344924Z OK
2022-11-23T02:59:56.2344947Z 
2022-11-23T02:59:56.2345069Z Generating XML reports...
2022-11-23T02:59:56.2345528Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025439.xml
2022-11-23T02:59:56.2345937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2346118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2346496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2346684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2346703Z 
2022-11-23T02:59:56.2346810Z Running tests...
2022-11-23T02:59:56.2347074Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2347379Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2347645Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2347841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91908
2022-11-23T02:59:56.2348060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91909
2022-11-23T02:59:56.2348421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2348594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2348966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2349152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2349514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2349685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2350127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2350296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2350524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2350766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2350989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2351229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2351623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2352015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2352120Z ok (3.927s)
2022-11-23T02:59:56.2352139Z 
2022-11-23T02:59:56.2352398Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2352492Z Ran 1 test in 3.927s
2022-11-23T02:59:56.2352511Z 
2022-11-23T02:59:56.2352605Z OK
2022-11-23T02:59:56.2352624Z 
2022-11-23T02:59:56.2352744Z Generating XML reports...
2022-11-23T02:59:56.2353196Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025448.xml
2022-11-23T02:59:56.2353564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2353737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2354112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2354301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2354323Z 
2022-11-23T02:59:56.2354414Z Running tests...
2022-11-23T02:59:56.2354673Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2355030Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2355311Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2355527Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92015
2022-11-23T02:59:56.2355741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92016
2022-11-23T02:59:56.2356110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2356283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2356639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2356829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2357187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2357359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2357728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2357913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2358136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2358673Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2358947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2359479Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2359731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zw3xqms
2022-11-23T02:59:56.2359982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zw3xqms/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2360232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexjmjrpz
2022-11-23T02:59:56.2360496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexjmjrpz/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2361040Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2361578Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2362111Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2362688Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2363225Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2363747Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2364280Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2364806Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2365375Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2365888Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2366415Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2366939Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2367474Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2368000Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2368108Z ok (6.743s)
2022-11-23T02:59:56.2368128Z 
2022-11-23T02:59:56.2368437Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2368550Z Ran 1 test in 6.743s
2022-11-23T02:59:56.2368570Z 
2022-11-23T02:59:56.2368663Z OK
2022-11-23T02:59:56.2368681Z 
2022-11-23T02:59:56.2368805Z Generating XML reports...
2022-11-23T02:59:56.2369265Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025454.xml
2022-11-23T02:59:56.2369615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2369789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2370162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2370358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2370377Z 
2022-11-23T02:59:56.2370485Z Running tests...
2022-11-23T02:59:56.2370750Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2371059Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2371350Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2371565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92136
2022-11-23T02:59:56.2371762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92137
2022-11-23T02:59:56.2372129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2372303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2372734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2372920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2373282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2373453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2373818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2373985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2374213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2374752Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2374985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2375511Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2375768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptv9xyqy_
2022-11-23T02:59:56.2376033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptv9xyqy_/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2376286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvstomks0
2022-11-23T02:59:56.2376554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvstomks0/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2377134Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2377675Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2378399Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2378946Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2379474Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2380101Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2380628Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2381152Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2381677Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2382193Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2382720Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2383298Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:59:56.2383847Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2384375Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:59:56.2384486Z ok (6.742s)
2022-11-23T02:59:56.2384507Z 
2022-11-23T02:59:56.2384780Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2384875Z Ran 1 test in 6.742s
2022-11-23T02:59:56.2384912Z 
2022-11-23T02:59:56.2384987Z OK
2022-11-23T02:59:56.2385006Z 
2022-11-23T02:59:56.2385128Z Generating XML reports...
2022-11-23T02:59:56.2385585Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025503.xml
2022-11-23T02:59:56.2385957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2386132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2386576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2386768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2386790Z 
2022-11-23T02:59:56.2386903Z Running tests...
2022-11-23T02:59:56.2387145Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2387454Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2387726Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2387942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92257
2022-11-23T02:59:56.2388156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92258
2022-11-23T02:59:56.2388526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2388706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2389090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2389279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2389623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2389866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2390490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2390727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2391094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2391327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2391581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3mgeldk9
2022-11-23T02:59:56.2391914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3mgeldk9/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2392155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa65xi73m
2022-11-23T02:59:56.2392418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa65xi73m/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2392653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2392887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2392989Z ok (8.127s)
2022-11-23T02:59:56.2393010Z 
2022-11-23T02:59:56.2393285Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2393403Z Ran 1 test in 8.128s
2022-11-23T02:59:56.2393423Z 
2022-11-23T02:59:56.2393514Z OK
2022-11-23T02:59:56.2393533Z 
2022-11-23T02:59:56.2393640Z Generating XML reports...
2022-11-23T02:59:56.2394099Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025512.xml
2022-11-23T02:59:56.2394476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2394801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2395219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2395410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2395430Z 
2022-11-23T02:59:56.2395537Z Running tests...
2022-11-23T02:59:56.2395795Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2396166Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2396428Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2396648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92378
2022-11-23T02:59:56.2396864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92379
2022-11-23T02:59:56.2397227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2397399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2397774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2397963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2398319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2398478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2398850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2399034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2399261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2399484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2399739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rvwrk5t
2022-11-23T02:59:56.2400003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rvwrk5t/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2400254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyb7pt0j3
2022-11-23T02:59:56.2400522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyb7pt0j3/_remote_module_non_scriptable.py
2022-11-23T02:59:56.2400735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2401049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:59:56.2401159Z ok (7.328s)
2022-11-23T02:59:56.2401179Z 
2022-11-23T02:59:56.2401445Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2401556Z Ran 1 test in 7.328s
2022-11-23T02:59:56.2401576Z 
2022-11-23T02:59:56.2401668Z OK
2022-11-23T02:59:56.2401687Z 
2022-11-23T02:59:56.2401811Z Generating XML reports...
2022-11-23T02:59:56.2402269Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025523.xml
2022-11-23T02:59:56.2402618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2402796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2403168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2403359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2403380Z 
2022-11-23T02:59:56.2403486Z Running tests...
2022-11-23T02:59:56.2403744Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2404051Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2404316Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2404532Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92499
2022-11-23T02:59:56.2404728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92500
2022-11-23T02:59:56.2404993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92501
2022-11-23T02:59:56.2405360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2405538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2405917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2406104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2406460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2406631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2406982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2407175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2407525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2407696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2408067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2408253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2408481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:59:56.2408705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2408922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2409053Z skip: Need at least 3 CUDA devices (3.880s)
2022-11-23T02:59:56.2409076Z 
2022-11-23T02:59:56.2409337Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2409449Z Ran 1 test in 3.880s
2022-11-23T02:59:56.2409468Z 
2022-11-23T02:59:56.2409576Z OK (skipped=1)
2022-11-23T02:59:56.2409597Z 
2022-11-23T02:59:56.2409721Z Generating XML reports...
2022-11-23T02:59:56.2410197Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025532.xml
2022-11-23T02:59:56.2410573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2410747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2411104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2411290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2411311Z 
2022-11-23T02:59:56.2411418Z Running tests...
2022-11-23T02:59:56.2411680Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2412096Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2412459Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2412681Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92636
2022-11-23T02:59:56.2412895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92637
2022-11-23T02:59:56.2413107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92638
2022-11-23T02:59:56.2413464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2413636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2414011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2414260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2414626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2414801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2415173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2415361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2415701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2415873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2416243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2416434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2416659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2416887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:59:56.2417110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2417263Z skip: Need at least 3 CUDA devices (4.026s)
2022-11-23T02:59:56.2417284Z 
2022-11-23T02:59:56.2417545Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2417641Z Ran 1 test in 4.026s
2022-11-23T02:59:56.2417661Z 
2022-11-23T02:59:56.2417770Z OK (skipped=1)
2022-11-23T02:59:56.2417790Z 
2022-11-23T02:59:56.2418097Z Generating XML reports...
2022-11-23T02:59:56.2418726Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025539.xml
2022-11-23T02:59:56.2419102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2419275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2419733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2419928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2419948Z 
2022-11-23T02:59:56.2420056Z Running tests...
2022-11-23T02:59:56.2420301Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2420613Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2420944Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: Frequently times out see https://github.com/pytorch/pytorch/issues/58920 (0.001s)
2022-11-23T02:59:56.2420964Z 
2022-11-23T02:59:56.2421223Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2421334Z Ran 1 test in 0.001s
2022-11-23T02:59:56.2421354Z 
2022-11-23T02:59:56.2421462Z OK (skipped=1)
2022-11-23T02:59:56.2421481Z 
2022-11-23T02:59:56.2421605Z Generating XML reports...
2022-11-23T02:59:56.2422036Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025545.xml
2022-11-23T02:59:56.2422386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2422559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2422932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2423120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2423139Z 
2022-11-23T02:59:56.2423252Z Running tests...
2022-11-23T02:59:56.2423580Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2423889Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2424159Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2424375Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92806
2022-11-23T02:59:56.2424571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92807
2022-11-23T02:59:56.2424783Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92808
2022-11-23T02:59:56.2425145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2425319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2425692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2425885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2426249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2426425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2426776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2426962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2427318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2427489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2427853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2428040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2428270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2428541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:59:56.2428770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2428902Z skip: Need at least 3 CUDA devices (3.997s)
2022-11-23T02:59:56.2428922Z 
2022-11-23T02:59:56.2429185Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2429297Z Ran 1 test in 3.997s
2022-11-23T02:59:56.2429316Z 
2022-11-23T02:59:56.2429422Z OK (skipped=1)
2022-11-23T02:59:56.2429442Z 
2022-11-23T02:59:56.2429567Z Generating XML reports...
2022-11-23T02:59:56.2429993Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025547.xml
2022-11-23T02:59:56.2430360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2430533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2430890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2431193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2431221Z 
2022-11-23T02:59:56.2431402Z Running tests...
2022-11-23T02:59:56.2431672Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2431979Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2432243Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2432460Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92943
2022-11-23T02:59:56.2432738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92944
2022-11-23T02:59:56.2432950Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92945
2022-11-23T02:59:56.2433303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2433474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2433904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2434211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2434580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2434750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2435118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2435310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2435654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2435823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2436194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2436378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2436602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2436826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:59:56.2437050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2437156Z ok (3.994s)
2022-11-23T02:59:56.2437177Z 
2022-11-23T02:59:56.2437439Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2437534Z Ran 1 test in 3.995s
2022-11-23T02:59:56.2437553Z 
2022-11-23T02:59:56.2437647Z OK
2022-11-23T02:59:56.2437666Z 
2022-11-23T02:59:56.2437846Z Generating XML reports...
2022-11-23T02:59:56.2438286Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025553.xml
2022-11-23T02:59:56.2438648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2438819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2439192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2439379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2439402Z 
2022-11-23T02:59:56.2439508Z Running tests...
2022-11-23T02:59:56.2439749Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2440057Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2440320Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2440533Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93080
2022-11-23T02:59:56.2440746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93081
2022-11-23T02:59:56.2440956Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93082
2022-11-23T02:59:56.2441320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2441491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2441849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2442090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2442451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2442623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2442988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2443174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2443527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2443700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2444046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2444236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2444460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2444682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2444903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:59:56.2445005Z ok (3.958s)
2022-11-23T02:59:56.2445025Z 
2022-11-23T02:59:56.2445287Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2445398Z Ran 1 test in 3.958s
2022-11-23T02:59:56.2445417Z 
2022-11-23T02:59:56.2445507Z OK
2022-11-23T02:59:56.2445526Z 
2022-11-23T02:59:56.2445632Z Generating XML reports...
2022-11-23T02:59:56.2446059Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025600.xml
2022-11-23T02:59:56.2446429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2446599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2447020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2447213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2447233Z 
2022-11-23T02:59:56.2447342Z Running tests...
2022-11-23T02:59:56.2447602Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2447893Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2448149Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2448365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93217
2022-11-23T02:59:56.2448583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93218
2022-11-23T02:59:56.2448793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93219
2022-11-23T02:59:56.2449161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2449333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2449707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2449894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2450240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2450410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2450778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2451028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2451388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2451560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2451927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2452111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2452318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:59:56.2452539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2452760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2452864Z ok (3.937s)
2022-11-23T02:59:56.2452884Z 
2022-11-23T02:59:56.2453149Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2453260Z Ran 1 test in 3.937s
2022-11-23T02:59:56.2453279Z 
2022-11-23T02:59:56.2453371Z OK
2022-11-23T02:59:56.2453393Z 
2022-11-23T02:59:56.2453515Z Generating XML reports...
2022-11-23T02:59:56.2453924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025606.xml
2022-11-23T02:59:56.2454285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2454456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2454824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2455008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2455031Z 
2022-11-23T02:59:56.2455138Z Running tests...
2022-11-23T02:59:56.2455400Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2455708Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2456012Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: Test does not pass when run locally (0.001s)
2022-11-23T02:59:56.2456032Z 
2022-11-23T02:59:56.2456276Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2456387Z Ran 1 test in 0.001s
2022-11-23T02:59:56.2456406Z 
2022-11-23T02:59:56.2456512Z OK (skipped=1)
2022-11-23T02:59:56.2456531Z 
2022-11-23T02:59:56.2456652Z Generating XML reports...
2022-11-23T02:59:56.2457073Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025612.xml
2022-11-23T02:59:56.2457434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2457608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2458158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2458359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2458379Z 
2022-11-23T02:59:56.2458468Z Running tests...
2022-11-23T02:59:56.2458730Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2459036Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2459272Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2459488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93387
2022-11-23T02:59:56.2459699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93388
2022-11-23T02:59:56.2459991Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93389
2022-11-23T02:59:56.2460361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2460520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2460892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2461076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2461436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2461608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2462166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2462379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2462743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2462911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2463268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2463451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2463676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2463895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2464116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:59:56.2464263Z skip: Need at least 3 CUDA devices (4.055s)
2022-11-23T02:59:56.2464288Z 
2022-11-23T02:59:56.2464553Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2464665Z Ran 1 test in 4.055s
2022-11-23T02:59:56.2464684Z 
2022-11-23T02:59:56.2464774Z OK (skipped=1)
2022-11-23T02:59:56.2464792Z 
2022-11-23T02:59:56.2464935Z Generating XML reports...
2022-11-23T02:59:56.2465440Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025615.xml
2022-11-23T02:59:56.2466009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2466185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2466562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2466748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2466768Z 
2022-11-23T02:59:56.2466876Z Running tests...
2022-11-23T02:59:56.2467140Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2467432Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2467759Z   test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2467976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93524
2022-11-23T02:59:56.2468337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2468509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2468877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2469064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2469288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2469577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2469979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2470083Z ok (5.339s)
2022-11-23T02:59:56.2470103Z 
2022-11-23T02:59:56.2470362Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2470472Z Ran 1 test in 5.339s
2022-11-23T02:59:56.2470491Z 
2022-11-23T02:59:56.2470583Z OK
2022-11-23T02:59:56.2470601Z 
2022-11-23T02:59:56.2470722Z Generating XML reports...
2022-11-23T02:59:56.2471271Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123025621.xml
2022-11-23T02:59:56.2471635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2471795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2472166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2472356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2472376Z 
2022-11-23T02:59:56.2472489Z Running tests...
2022-11-23T02:59:56.2472744Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2473051Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2473385Z   test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2473604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93599
2022-11-23T02:59:56.2473956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2474131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2474500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2474735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2474966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2475207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2475604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2476345Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:59:56.2476462Z   warnings.warn(
2022-11-23T02:59:56.2476545Z ok (5.196s)
2022-11-23T02:59:56.2476582Z 
2022-11-23T02:59:56.2476829Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2476946Z Ran 1 test in 5.196s
2022-11-23T02:59:56.2476965Z 
2022-11-23T02:59:56.2477056Z OK
2022-11-23T02:59:56.2477075Z 
2022-11-23T02:59:56.2477198Z Generating XML reports...
2022-11-23T02:59:56.2477747Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123025629.xml
2022-11-23T02:59:56.2478111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2478284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2478661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2478884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2478919Z 
2022-11-23T02:59:56.2479010Z Running tests...
2022-11-23T02:59:56.2479272Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2479583Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2479944Z   test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2480161Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93674
2022-11-23T02:59:56.2480527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2480698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2481070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2481245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2481471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2481717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2482115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2482217Z ok (5.352s)
2022-11-23T02:59:56.2482237Z 
2022-11-23T02:59:56.2482496Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2482609Z Ran 1 test in 5.352s
2022-11-23T02:59:56.2482629Z 
2022-11-23T02:59:56.2482720Z OK
2022-11-23T02:59:56.2482739Z 
2022-11-23T02:59:56.2482845Z Generating XML reports...
2022-11-23T02:59:56.2483390Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123025636.xml
2022-11-23T02:59:56.2483760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2483933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2484356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2484548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2484567Z 
2022-11-23T02:59:56.2484675Z Running tests...
2022-11-23T02:59:56.2484935Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2485244Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2485557Z   test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2485776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93749
2022-11-23T02:59:56.2486141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2486316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2486687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2486874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2487095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2487335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2487715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2487818Z ok (5.307s)
2022-11-23T02:59:56.2487886Z 
2022-11-23T02:59:56.2488154Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2488264Z Ran 1 test in 5.307s
2022-11-23T02:59:56.2488283Z 
2022-11-23T02:59:56.2488374Z OK
2022-11-23T02:59:56.2488392Z 
2022-11-23T02:59:56.2488517Z Generating XML reports...
2022-11-23T02:59:56.2489066Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123025644.xml
2022-11-23T02:59:56.2489430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2489604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2489962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2490147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2490166Z 
2022-11-23T02:59:56.2490280Z Running tests...
2022-11-23T02:59:56.2490542Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2490845Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2491107Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: GPUs are available, skipping test (0.001s)
2022-11-23T02:59:56.2491127Z 
2022-11-23T02:59:56.2491384Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2491495Z Ran 1 test in 0.001s
2022-11-23T02:59:56.2491513Z 
2022-11-23T02:59:56.2491620Z OK (skipped=1)
2022-11-23T02:59:56.2491639Z 
2022-11-23T02:59:56.2491743Z Generating XML reports...
2022-11-23T02:59:56.2492190Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123025652.xml
2022-11-23T02:59:56.2492556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2492733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2493106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2493342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2493362Z 
2022-11-23T02:59:56.2493472Z Running tests...
2022-11-23T02:59:56.2493734Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2494025Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2494278Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2494492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93857
2022-11-23T02:59:56.2494708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93858
2022-11-23T02:59:56.2495076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2495249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2495660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2495835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2496212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2496382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2496759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2496943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2497171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2497464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2497683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2498142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2498558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2498951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2499036Z ok (5.471s)
2022-11-23T02:59:56.2499056Z 
2022-11-23T02:59:56.2499315Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2499426Z Ran 1 test in 5.471s
2022-11-23T02:59:56.2499446Z 
2022-11-23T02:59:56.2499538Z OK
2022-11-23T02:59:56.2499561Z 
2022-11-23T02:59:56.2499685Z Generating XML reports...
2022-11-23T02:59:56.2500112Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025654.xml
2022-11-23T02:59:56.2500482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2500657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2501015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2501202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2501221Z 
2022-11-23T02:59:56.2501327Z Running tests...
2022-11-23T02:59:56.2501585Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2501893Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2502146Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2502364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93966
2022-11-23T02:59:56.2502654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93967
2022-11-23T02:59:56.2503012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2503188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2503559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2503746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2504105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2504275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2504648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2504833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2505063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2505285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2505507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2505739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2506136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2506529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2506696Z ok (6.651s)
2022-11-23T02:59:56.2506716Z 
2022-11-23T02:59:56.2506981Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2507093Z Ran 1 test in 6.652s
2022-11-23T02:59:56.2507112Z 
2022-11-23T02:59:56.2507206Z OK
2022-11-23T02:59:56.2507226Z 
2022-11-23T02:59:56.2507332Z Generating XML reports...
2022-11-23T02:59:56.2507758Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025702.xml
2022-11-23T02:59:56.2508121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2508294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2508670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2508857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2508880Z 
2022-11-23T02:59:56.2508989Z Running tests...
2022-11-23T02:59:56.2509250Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2509544Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2509784Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2510001Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94083
2022-11-23T02:59:56.2510212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94084
2022-11-23T02:59:56.2510575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2510746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2511118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2511308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2511665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2511861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2512248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2512430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2512655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2512896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2513114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2513351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2513745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2514117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2514220Z ok (6.661s)
2022-11-23T02:59:56.2514240Z 
2022-11-23T02:59:56.2514500Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2514611Z Ran 1 test in 6.661s
2022-11-23T02:59:56.2514631Z 
2022-11-23T02:59:56.2514720Z OK
2022-11-23T02:59:56.2514739Z 
2022-11-23T02:59:56.2514862Z Generating XML reports...
2022-11-23T02:59:56.2515289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025711.xml
2022-11-23T02:59:56.2515651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2515888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2516248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2516437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2516457Z 
2022-11-23T02:59:56.2516567Z Running tests...
2022-11-23T02:59:56.2516827Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2517133Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2517371Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2517587Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94200
2022-11-23T02:59:56.2517801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94201
2022-11-23T02:59:56.2518155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2518328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2518700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2518886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2519243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2519415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2519781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2519963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2520186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2520412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2520694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2520939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2521339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2521729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2521832Z ok (6.644s)
2022-11-23T02:59:56.2521851Z 
2022-11-23T02:59:56.2522111Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2522224Z Ran 1 test in 6.645s
2022-11-23T02:59:56.2522243Z 
2022-11-23T02:59:56.2522321Z OK
2022-11-23T02:59:56.2522358Z 
2022-11-23T02:59:56.2522463Z Generating XML reports...
2022-11-23T02:59:56.2522892Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025720.xml
2022-11-23T02:59:56.2523257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2523428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2523802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2523991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2524010Z 
2022-11-23T02:59:56.2524120Z Running tests...
2022-11-23T02:59:56.2524378Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2524669Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2524954Z   test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2525169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94317
2022-11-23T02:59:56.2525385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94318
2022-11-23T02:59:56.2525752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2525925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2526298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2526483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2526826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2526995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2527370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2527554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2527782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2528020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2528246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2528484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2528882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2529254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2529359Z ok (6.504s)
2022-11-23T02:59:56.2529378Z 
2022-11-23T02:59:56.2529642Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2529753Z Ran 1 test in 6.504s
2022-11-23T02:59:56.2529817Z 
2022-11-23T02:59:56.2529913Z OK
2022-11-23T02:59:56.2529931Z 
2022-11-23T02:59:56.2530056Z Generating XML reports...
2022-11-23T02:59:56.2530486Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025729.xml
2022-11-23T02:59:56.2530850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2531023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2531380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2531571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2531591Z 
2022-11-23T02:59:56.2531700Z Running tests...
2022-11-23T02:59:56.2531958Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2532269Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2532512Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2532727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94434
2022-11-23T02:59:56.2532942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94435
2022-11-23T02:59:56.2533289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2533463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2533834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2534073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2534433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2534606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2534977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2535163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2535390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2535616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2535834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2536069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2536465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2536858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2536960Z ok (6.700s)
2022-11-23T02:59:56.2536979Z 
2022-11-23T02:59:56.2537243Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2537353Z Ran 1 test in 6.700s
2022-11-23T02:59:56.2537372Z 
2022-11-23T02:59:56.2537446Z OK
2022-11-23T02:59:56.2537481Z 
2022-11-23T02:59:56.2537587Z Generating XML reports...
2022-11-23T02:59:56.2538331Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025737.xml
2022-11-23T02:59:56.2538712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2538895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2539339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2539533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2539553Z 
2022-11-23T02:59:56.2539662Z Running tests...
2022-11-23T02:59:56.2539924Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2540217Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2540455Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2540670Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94551
2022-11-23T02:59:56.2540883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94552
2022-11-23T02:59:56.2541249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2541423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2541799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2541985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2542330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2542500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2542867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2543052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2543362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2543604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2543831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2544070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2544467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2544841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2544943Z ok (6.657s)
2022-11-23T02:59:56.2544962Z 
2022-11-23T02:59:56.2545222Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2545333Z Ran 1 test in 6.658s
2022-11-23T02:59:56.2545356Z 
2022-11-23T02:59:56.2545448Z OK
2022-11-23T02:59:56.2545467Z 
2022-11-23T02:59:56.2545591Z Generating XML reports...
2022-11-23T02:59:56.2546017Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025746.xml
2022-11-23T02:59:56.2546383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2546538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2546915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2547103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2547123Z 
2022-11-23T02:59:56.2547230Z Running tests...
2022-11-23T02:59:56.2547490Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2547797Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2548042Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2548259Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94668
2022-11-23T02:59:56.2548518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94669
2022-11-23T02:59:56.2548874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2549047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2549421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2549607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2549965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2550140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2550506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2550694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2550901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2551142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2551360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2551595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2551991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2552384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2552531Z ok (5.451s)
2022-11-23T02:59:56.2552551Z 
2022-11-23T02:59:56.2552813Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2552927Z Ran 1 test in 5.452s
2022-11-23T02:59:56.2552947Z 
2022-11-23T02:59:56.2553020Z OK
2022-11-23T02:59:56.2553040Z 
2022-11-23T02:59:56.2553163Z Generating XML reports...
2022-11-23T02:59:56.2553590Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025755.xml
2022-11-23T02:59:56.2553957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2554131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2554507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2554698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2554717Z 
2022-11-23T02:59:56.2554826Z Running tests...
2022-11-23T02:59:56.2555085Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2555379Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2555610Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2555825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94777
2022-11-23T02:59:56.2556038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94778
2022-11-23T02:59:56.2556399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2556573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2556949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2557139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2557521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2557698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2558073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2558262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2558486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2558727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2558947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2559189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2559588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2559964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2560067Z ok (6.641s)
2022-11-23T02:59:56.2560086Z 
2022-11-23T02:59:56.2560346Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2560457Z Ran 1 test in 6.642s
2022-11-23T02:59:56.2560477Z 
2022-11-23T02:59:56.2560569Z OK
2022-11-23T02:59:56.2560588Z 
2022-11-23T02:59:56.2560711Z Generating XML reports...
2022-11-23T02:59:56.2561138Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025803.xml
2022-11-23T02:59:56.2561504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2561705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2562082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2562271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2562291Z 
2022-11-23T02:59:56.2562399Z Running tests...
2022-11-23T02:59:56.2562660Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2562970Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2563209Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2563423Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94898
2022-11-23T02:59:56.2563637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94899
2022-11-23T02:59:56.2563992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2564164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2564538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2564724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2565080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2565250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2565618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2565804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2566017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2566255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2566523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2566761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2567157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2567547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2567651Z ok (11.243s)
2022-11-23T02:59:56.2567672Z 
2022-11-23T02:59:56.2567933Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2568048Z Ran 1 test in 11.243s
2022-11-23T02:59:56.2568070Z 
2022-11-23T02:59:56.2568145Z OK
2022-11-23T02:59:56.2568164Z 
2022-11-23T02:59:56.2568287Z Generating XML reports...
2022-11-23T02:59:56.2568712Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025812.xml
2022-11-23T02:59:56.2569081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2569253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2569627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2569815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2569835Z 
2022-11-23T02:59:56.2569943Z Running tests...
2022-11-23T02:59:56.2570188Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2570499Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2570797Z   test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2571016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95019
2022-11-23T02:59:56.2571231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95020
2022-11-23T02:59:56.2571600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2571774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2572152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2572340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2572684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2572856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2573227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2573417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2573641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2573883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2574104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2574339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2574718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2575112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2575217Z ok (5.426s)
2022-11-23T02:59:56.2575236Z 
2022-11-23T02:59:56.2575532Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2575649Z Ran 1 test in 5.426s
2022-11-23T02:59:56.2575668Z 
2022-11-23T02:59:56.2575760Z OK
2022-11-23T02:59:56.2575778Z 
2022-11-23T02:59:56.2575901Z Generating XML reports...
2022-11-23T02:59:56.2576330Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025826.xml
2022-11-23T02:59:56.2576698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2576855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2577232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2577423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2577442Z 
2022-11-23T02:59:56.2577550Z Running tests...
2022-11-23T02:59:56.2577811Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2578354Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2578592Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2578807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95131
2022-11-23T02:59:56.2579003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95132
2022-11-23T02:59:56.2579374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2579547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2580045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2580234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2580597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2580767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2581136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2581319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2581525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2581765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2581989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2582235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2582852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2583251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2583352Z ok (6.593s)
2022-11-23T02:59:56.2583372Z 
2022-11-23T02:59:56.2583632Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2583742Z Ran 1 test in 6.593s
2022-11-23T02:59:56.2583762Z 
2022-11-23T02:59:56.2583836Z OK
2022-11-23T02:59:56.2583855Z 
2022-11-23T02:59:56.2583978Z Generating XML reports...
2022-11-23T02:59:56.2584405Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025834.xml
2022-11-23T02:59:56.2584774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2584946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2585398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2585594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2585614Z 
2022-11-23T02:59:56.2585724Z Running tests...
2022-11-23T02:59:56.2585972Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2586281Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2586538Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2586752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95248
2022-11-23T02:59:56.2586971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95249
2022-11-23T02:59:56.2587337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2587516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2587889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2588075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2588418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2588588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2588955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2589141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2589411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2589650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2589875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2590106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2590488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2590880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2590984Z ok (5.446s)
2022-11-23T02:59:56.2591004Z 
2022-11-23T02:59:56.2591263Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2591379Z Ran 1 test in 5.446s
2022-11-23T02:59:56.2591399Z 
2022-11-23T02:59:56.2591490Z OK
2022-11-23T02:59:56.2591509Z 
2022-11-23T02:59:56.2591631Z Generating XML reports...
2022-11-23T02:59:56.2592058Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025843.xml
2022-11-23T02:59:56.2592421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2592576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2592948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2593136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2593155Z 
2022-11-23T02:59:56.2593264Z Running tests...
2022-11-23T02:59:56.2593524Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2593835Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2594091Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2594346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95357
2022-11-23T02:59:56.2594550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95358
2022-11-23T02:59:56.2594918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2595089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2595466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2595655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2596014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2596189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2596566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2596751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2596962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2597202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2597422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2597654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2598051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2598494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2598597Z ok (6.558s)
2022-11-23T02:59:56.2598616Z 
2022-11-23T02:59:56.2598879Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2598991Z Ran 1 test in 6.558s
2022-11-23T02:59:56.2599012Z 
2022-11-23T02:59:56.2599086Z OK
2022-11-23T02:59:56.2599104Z 
2022-11-23T02:59:56.2599227Z Generating XML reports...
2022-11-23T02:59:56.2599655Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025850.xml
2022-11-23T02:59:56.2600020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2600193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2600564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2600754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2600773Z 
2022-11-23T02:59:56.2600881Z Running tests...
2022-11-23T02:59:56.2601124Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2601432Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2601680Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2601897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95474
2022-11-23T02:59:56.2602110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95475
2022-11-23T02:59:56.2602472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2602645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2603020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2603207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2603605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2603782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2604154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2604337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2604564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2604805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2605033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2605272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2605652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2606042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2606142Z ok (6.653s)
2022-11-23T02:59:56.2606161Z 
2022-11-23T02:59:56.2606421Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2606533Z Ran 1 test in 6.653s
2022-11-23T02:59:56.2606552Z 
2022-11-23T02:59:56.2606643Z OK
2022-11-23T02:59:56.2606661Z 
2022-11-23T02:59:56.2606784Z Generating XML reports...
2022-11-23T02:59:56.2607211Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025859.xml
2022-11-23T02:59:56.2607637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2607792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2608169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2608357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2608377Z 
2022-11-23T02:59:56.2608483Z Running tests...
2022-11-23T02:59:56.2608742Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2609055Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2609299Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2609515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95591
2022-11-23T02:59:56.2609717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95592
2022-11-23T02:59:56.2610082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2610255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2610629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2610815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2611173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2611340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2611705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2611893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2612101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2612385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2612612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2612849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2613243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2613635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2613738Z ok (5.430s)
2022-11-23T02:59:56.2613758Z 
2022-11-23T02:59:56.2614016Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2614113Z Ran 1 test in 5.430s
2022-11-23T02:59:56.2614147Z 
2022-11-23T02:59:56.2614221Z OK
2022-11-23T02:59:56.2614241Z 
2022-11-23T02:59:56.2614364Z Generating XML reports...
2022-11-23T02:59:56.2614794Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025908.xml
2022-11-23T02:59:56.2615161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2615334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2615707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2615893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2615912Z 
2022-11-23T02:59:56.2616021Z Running tests...
2022-11-23T02:59:56.2616265Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2616630Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2616864Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2617080Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95700
2022-11-23T02:59:56.2617293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95701
2022-11-23T02:59:56.2617658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2617831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2618444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2618615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2618979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2619147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2619519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2619703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2619934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2620175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2620399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2620637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2621017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2621413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2621515Z ok (6.655s)
2022-11-23T02:59:56.2621534Z 
2022-11-23T02:59:56.2621861Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2621977Z Ran 1 test in 6.655s
2022-11-23T02:59:56.2621996Z 
2022-11-23T02:59:56.2622088Z OK
2022-11-23T02:59:56.2622106Z 
2022-11-23T02:59:56.2622231Z Generating XML reports...
2022-11-23T02:59:56.2622662Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025916.xml
2022-11-23T02:59:56.2623028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2623183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2623562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2623748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2623768Z 
2022-11-23T02:59:56.2623877Z Running tests...
2022-11-23T02:59:56.2624138Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2624443Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2624683Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2624896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95821
2022-11-23T02:59:56.2625092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95822
2022-11-23T02:59:56.2625459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2625697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2626073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2626259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2626622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2626792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2627159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2627342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2627550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2627791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2628018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2628255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2628654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2629045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2629147Z ok (11.255s)
2022-11-23T02:59:56.2629167Z 
2022-11-23T02:59:56.2629425Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2629520Z Ran 1 test in 11.255s
2022-11-23T02:59:56.2629555Z 
2022-11-23T02:59:56.2629628Z OK
2022-11-23T02:59:56.2629647Z 
2022-11-23T02:59:56.2629769Z Generating XML reports...
2022-11-23T02:59:56.2630194Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025925.xml
2022-11-23T02:59:56.2630561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2630734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2631149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2631342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2631361Z 
2022-11-23T02:59:56.2631468Z Running tests...
2022-11-23T02:59:56.2631713Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2632022Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2632256Z   test_send_recv (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2632474Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95942
2022-11-23T02:59:56.2632690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95943
2022-11-23T02:59:56.2633057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2633231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2633606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2633774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2634134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2634304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2634674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2634906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2635129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:59:56.2635372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2635597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:59:56.2635836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:59:56.2636217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2636604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:59:56.2636705Z ok (5.500s)
2022-11-23T02:59:56.2636724Z 
2022-11-23T02:59:56.2636986Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2637097Z Ran 1 test in 5.500s
2022-11-23T02:59:56.2637116Z 
2022-11-23T02:59:56.2637208Z OK
2022-11-23T02:59:56.2637227Z 
2022-11-23T02:59:56.2637347Z Generating XML reports...
2022-11-23T02:59:56.2637777Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025939.xml
2022-11-23T02:59:56.2638125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2638296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2638670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2638858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2638877Z 
2022-11-23T02:59:56.2638984Z Running tests...
2022-11-23T02:59:56.2639250Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2639558Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2639784Z   test_common_errors (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2640068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2640452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2640692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2641084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2641318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2641704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2641942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2642335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2642436Z ok (1.645s)
2022-11-23T02:59:56.2642455Z 
2022-11-23T02:59:56.2642713Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2642806Z Ran 1 test in 1.645s
2022-11-23T02:59:56.2642825Z 
2022-11-23T02:59:56.2642917Z OK
2022-11-23T02:59:56.2642936Z 
2022-11-23T02:59:56.2643057Z Generating XML reports...
2022-11-23T02:59:56.2643469Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123025946.xml
2022-11-23T02:59:56.2643833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:59:56.2644068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:59:56.2644447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:59:56.2644638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:59:56.2644657Z 
2022-11-23T02:59:56.2644764Z Running tests...
2022-11-23T02:59:56.2645007Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2645317Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:59:56.2645550Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:59:56.2645788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2646183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2646426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:59:56.2646818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:59:56.2646921Z ok (4.595s)
2022-11-23T02:59:56.2646940Z 
2022-11-23T02:59:56.2647180Z ----------------------------------------------------------------------
2022-11-23T02:59:56.2647289Z Ran 1 test in 4.596s
2022-11-23T02:59:56.2647308Z 
2022-11-23T02:59:56.2647399Z OK
2022-11-23T02:59:56.2647418Z 
2022-11-23T02:59:56.2647543Z Generating XML reports...
2022-11-23T02:59:56.2647934Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123025950.xml
2022-11-23T02:59:56.2647953Z 
2022-11-23T02:59:56.2648454Z ##[endgroup]
2022-11-23T02:59:56.2648882Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_fzn4qwas)
2022-11-23T02:59:56.2648906Z 
2022-11-23T02:59:56.4529864Z 
2022-11-23T02:59:56.4530411Z real	98m30.589s
2022-11-23T02:59:56.4530789Z user	161m20.038s
2022-11-23T02:59:56.4531006Z sys	84m35.610s
2022-11-23T02:59:56.4531546Z + assert_git_not_dirty
2022-11-23T02:59:56.4531991Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-11-23T02:59:56.4532230Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *xla* ]]
2022-11-23T02:59:56.4534019Z ++ git status --porcelain
2022-11-23T02:59:57.5158831Z + git_status=
2022-11-23T02:59:57.5159253Z + [[ -n '' ]]
2022-11-23T02:59:57.5159661Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-11-23T02:59:57.5159962Z + [[ 2 == 1 ]]
2022-11-23T02:59:57.5160166Z + [[ 2 == 1 ]]
2022-11-23T02:59:57.5231466Z Prepare all required actions
2022-11-23T02:59:57.5231900Z Getting action download info
2022-11-23T02:59:57.6689048Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2022-11-23T02:59:57.7963034Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-11-23T02:59:57.7963325Z with:
2022-11-23T02:59:57.7963737Z   github-token: ***
2022-11-23T02:59:57.7963975Z env:
2022-11-23T02:59:57.7964194Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:57.7964457Z   GPU_FLAG: --gpus all
2022-11-23T02:59:57.7964814Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T02:59:57.7965137Z ##[endgroup]
2022-11-23T02:59:57.7997478Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2022-11-23T02:59:57.7997793Z with:
2022-11-23T02:59:57.7997996Z   shell: bash
2022-11-23T02:59:57.7998234Z   timeout_minutes: 10
2022-11-23T02:59:57.7998480Z   max_attempts: 5
2022-11-23T02:59:57.7998753Z   retry_wait_seconds: 30
2022-11-23T02:59:57.7999242Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2022-11-23T02:59:57.7999873Z   polling_interval_seconds: 1
2022-11-23T02:59:57.8000145Z   warning_on_retry: true
2022-11-23T02:59:57.8000404Z   continue_on_error: false
2022-11-23T02:59:57.8000628Z env:
2022-11-23T02:59:57.8000861Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:57.8001126Z   GPU_FLAG: --gpus all
2022-11-23T02:59:57.8001466Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T02:59:57.8001947Z   GITHUB_TOKEN: ***
2022-11-23T02:59:57.8002194Z ##[endgroup]
2022-11-23T02:59:57.8682448Z + python3 -m pip install requests==2.26.0
2022-11-23T02:59:58.1517610Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:59:58.3109743Z Collecting requests==2.26.0
2022-11-23T02:59:58.3360854Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-11-23T02:59:58.3961493Z Collecting idna<4,>=2.5; python_version >= "3"
2022-11-23T02:59:58.4071895Z   Downloading idna-3.4-py3-none-any.whl (61 kB)
2022-11-23T02:59:58.5087267Z Collecting urllib3<1.27,>=1.21.1
2022-11-23T02:59:58.5130093Z   Downloading urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
2022-11-23T02:59:58.7114253Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2022-11-23T02:59:58.7184393Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2022-11-23T02:59:58.7838385Z Collecting certifi>=2017.4.17
2022-11-23T02:59:58.7905042Z   Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
2022-11-23T02:59:58.8774999Z Installing collected packages: idna, urllib3, charset-normalizer, certifi, requests
2022-11-23T02:59:59.0687054Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T02:59:59.0687733Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T02:59:59.1297231Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.26.0 urllib3-1.26.12
2022-11-23T02:59:59.1773605Z ++ python3 .github/scripts/get_workflow_job_id.py 3528293562 i-0a212d9242243d3a8
2022-11-23T03:00:00.8362332Z + GHA_WORKFLOW_JOB_ID=9655200205
2022-11-23T03:00:00.8362930Z + echo job-id=9655200205
2022-11-23T03:00:00.8676277Z Command completed after 1 attempt(s).
2022-11-23T03:00:00.8809736Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-11-23T03:00:00.8810066Z kill "$MONITOR_SCRIPT_PID"
2022-11-23T03:00:00.8823800Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:00:00.8824101Z env:
2022-11-23T03:00:00.8824326Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:00.8824593Z   GPU_FLAG: --gpus all
2022-11-23T03:00:00.8824954Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:00.8825299Z   MONITOR_SCRIPT_PID: 58981
2022-11-23T03:00:00.8825557Z ##[endgroup]
2022-11-23T03:00:00.8922357Z Prepare all required actions
2022-11-23T03:00:00.8922736Z Getting action download info
2022-11-23T03:00:01.1366376Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb)
2022-11-23T03:00:01.3185802Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-11-23T03:00:01.3186117Z with:
2022-11-23T03:00:01.3186469Z   file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205
2022-11-23T03:00:01.3186793Z env:
2022-11-23T03:00:01.3187030Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:01.3187293Z   GPU_FLAG: --gpus all
2022-11-23T03:00:01.3187633Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:01.3187977Z ##[endgroup]
2022-11-23T03:00:01.3218306Z ##[group]Run # Remove any previous test jsons if they exist
2022-11-23T03:00:01.3218772Z # Remove any previous test jsons if they exist
2022-11-23T03:00:01.3219089Z rm -f test-jsons-*.zip
2022-11-23T03:00:01.3219463Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-11-23T03:00:01.3232813Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:00:01.3233111Z env:
2022-11-23T03:00:01.3233355Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:01.3233602Z   GPU_FLAG: --gpus all
2022-11-23T03:00:01.3233969Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:01.3234438Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205
2022-11-23T03:00:01.3234773Z ##[endgroup]
2022-11-23T03:00:01.3386334Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2022-11-23T03:00:01.3421218Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-11-23T03:00:01.3428496Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-11-23T03:00:01.3429716Z   adding: test/.pytorch-slow-tests.json (deflated 73%)
2022-11-23T03:00:01.3440518Z   adding: test/.pytorch-disabled-tests.json (deflated 86%)
2022-11-23T03:00:01.3464130Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T03:00:01.3464524Z # Remove any previous test reports if they exist
2022-11-23T03:00:01.3464854Z rm -f test-reports-*.zip
2022-11-23T03:00:01.3465209Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2022-11-23T03:00:01.3477470Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:00:01.3477766Z env:
2022-11-23T03:00:01.3478007Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:01.3478254Z   GPU_FLAG: --gpus all
2022-11-23T03:00:01.3478609Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:01.3479079Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205
2022-11-23T03:00:01.3479484Z ##[endgroup]
2022-11-23T03:00:01.3632419Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012134.xml (deflated 43%)
2022-11-23T03:00:01.3633343Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012136.xml (deflated 44%)
2022-11-23T03:00:01.3634247Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012138.xml (deflated 42%)
2022-11-23T03:00:01.3635135Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012147.xml (deflated 42%)
2022-11-23T03:00:01.3636002Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012156.xml (deflated 42%)
2022-11-23T03:00:01.3636891Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012205.xml (deflated 42%)
2022-11-23T03:00:01.3637924Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012216.xml (deflated 43%)
2022-11-23T03:00:01.3638831Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012218.xml (deflated 44%)
2022-11-23T03:00:01.3639718Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012220.xml (deflated 42%)
2022-11-23T03:00:01.3640584Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012229.xml (deflated 42%)
2022-11-23T03:00:01.3641459Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012237.xml (deflated 42%)
2022-11-23T03:00:01.3642336Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012246.xml (deflated 42%)
2022-11-23T03:00:01.3643171Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012258.xml (deflated 41%)
2022-11-23T03:00:01.3644059Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012307.xml (deflated 41%)
2022-11-23T03:00:01.3644848Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012310.xml (deflated 42%)
2022-11-23T03:00:01.3645629Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012317.xml (deflated 42%)
2022-11-23T03:00:01.3646416Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012321.xml (deflated 42%)
2022-11-23T03:00:01.3647176Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012323.xml (deflated 42%)
2022-11-23T03:00:01.3647969Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012326.xml (deflated 40%)
2022-11-23T03:00:01.3648762Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012336.xml (deflated 40%)
2022-11-23T03:00:01.3649539Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012345.xml (deflated 40%)
2022-11-23T03:00:01.3650326Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012355.xml (deflated 39%)
2022-11-23T03:00:01.3651089Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012405.xml (deflated 39%)
2022-11-23T03:00:01.3651858Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012415.xml (deflated 40%)
2022-11-23T03:00:01.3652642Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012425.xml (deflated 40%)
2022-11-23T03:00:01.3653421Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012434.xml (deflated 42%)
2022-11-23T03:00:01.3654193Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012439.xml (deflated 41%)
2022-11-23T03:00:01.3654968Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012446.xml (deflated 42%)
2022-11-23T03:00:01.3655747Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012450.xml (deflated 42%)
2022-11-23T03:00:01.3656522Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012459.xml (deflated 41%)
2022-11-23T03:00:01.3657361Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012502.xml (deflated 46%)
2022-11-23T03:00:01.3658372Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012512.xml (deflated 47%)
2022-11-23T03:00:01.3659170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012522.xml (deflated 48%)
2022-11-23T03:00:01.3659950Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012532.xml (deflated 45%)
2022-11-23T03:00:01.3660728Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012541.xml (deflated 41%)
2022-11-23T03:00:01.3661493Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012544.xml (deflated 43%)
2022-11-23T03:00:01.3663060Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012546.xml (deflated 43%)
2022-11-23T03:00:01.3664756Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012548.xml (deflated 43%)
2022-11-23T03:00:01.3666606Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012551.xml (deflated 44%)
2022-11-23T03:00:01.3668301Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012553.xml (deflated 43%)
2022-11-23T03:00:01.3669979Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012556.xml (deflated 41%)
2022-11-23T03:00:01.3671629Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012558.xml (deflated 40%)
2022-11-23T03:00:01.3673283Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012608.xml (deflated 40%)
2022-11-23T03:00:01.3674902Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012619.xml (deflated 41%)
2022-11-23T03:00:01.3676563Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012621.xml (deflated 41%)
2022-11-23T03:00:01.3678182Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012623.xml (deflated 40%)
2022-11-23T03:00:01.3679864Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012634.xml (deflated 40%)
2022-11-23T03:00:01.3681516Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012644.xml (deflated 42%)
2022-11-23T03:00:01.3683177Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012648.xml (deflated 42%)
2022-11-23T03:00:01.3684836Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012652.xml (deflated 40%)
2022-11-23T03:00:01.3686459Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012702.xml (deflated 40%)
2022-11-23T03:00:01.3688099Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012712.xml (deflated 40%)
2022-11-23T03:00:01.3689750Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012722.xml (deflated 41%)
2022-11-23T03:00:01.3691400Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012724.xml (deflated 41%)
2022-11-23T03:00:01.3693002Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012726.xml (deflated 41%)
2022-11-23T03:00:01.3694819Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012729.xml (deflated 41%)
2022-11-23T03:00:01.3696497Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012731.xml (deflated 41%)
2022-11-23T03:00:01.3698431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012734.xml (deflated 41%)
2022-11-23T03:00:01.3700038Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012736.xml (deflated 41%)
2022-11-23T03:00:01.3701668Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012738.xml (deflated 41%)
2022-11-23T03:00:01.3703295Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012741.xml (deflated 41%)
2022-11-23T03:00:01.3704967Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012743.xml (deflated 42%)
2022-11-23T03:00:01.3706606Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012746.xml (deflated 41%)
2022-11-23T03:00:01.3708399Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012748.xml (deflated 41%)
2022-11-23T03:00:01.3710021Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012750.xml (deflated 41%)
2022-11-23T03:00:01.3711664Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012753.xml (deflated 42%)
2022-11-23T03:00:01.3713277Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012755.xml (deflated 41%)
2022-11-23T03:00:01.3714915Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012757.xml (deflated 41%)
2022-11-23T03:00:01.3716558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012800.xml (deflated 41%)
2022-11-23T03:00:01.3718216Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012802.xml (deflated 41%)
2022-11-23T03:00:01.3719871Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012805.xml (deflated 41%)
2022-11-23T03:00:01.3721495Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012807.xml (deflated 41%)
2022-11-23T03:00:01.3723115Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012809.xml (deflated 41%)
2022-11-23T03:00:01.3724725Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012812.xml (deflated 42%)
2022-11-23T03:00:01.3726356Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012814.xml (deflated 42%)
2022-11-23T03:00:01.3727986Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012816.xml (deflated 42%)
2022-11-23T03:00:01.3729640Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012819.xml (deflated 43%)
2022-11-23T03:00:01.3731239Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012821.xml (deflated 43%)
2022-11-23T03:00:01.3732865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012824.xml (deflated 41%)
2022-11-23T03:00:01.3734530Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012826.xml (deflated 41%)
2022-11-23T03:00:01.3736320Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012835.xml (deflated 41%)
2022-11-23T03:00:01.3738215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012838.xml (deflated 42%)
2022-11-23T03:00:01.3739881Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012840.xml (deflated 41%)
2022-11-23T03:00:01.3741476Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012843.xml (deflated 40%)
2022-11-23T03:00:01.3743112Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012851.xml (deflated 40%)
2022-11-23T03:00:01.3744747Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012900.xml (deflated 41%)
2022-11-23T03:00:01.3746373Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012909.xml (deflated 42%)
2022-11-23T03:00:01.3748187Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012912.xml (deflated 42%)
2022-11-23T03:00:01.3749815Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012914.xml (deflated 41%)
2022-11-23T03:00:01.3751466Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012924.xml (deflated 41%)
2022-11-23T03:00:01.3753148Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012933.xml (deflated 42%)
2022-11-23T03:00:01.3754802Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012936.xml (deflated 41%)
2022-11-23T03:00:01.3756424Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012945.xml (deflated 42%)
2022-11-23T03:00:01.3758051Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012948.xml (deflated 42%)
2022-11-23T03:00:01.3759704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012955.xml (deflated 42%)
2022-11-23T03:00:01.3761325Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012957.xml (deflated 42%)
2022-11-23T03:00:01.3762922Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012959.xml (deflated 42%)
2022-11-23T03:00:01.3764543Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013003.xml (deflated 42%)
2022-11-23T03:00:01.3766167Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013014.xml (deflated 42%)
2022-11-23T03:00:01.3767789Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013016.xml (deflated 42%)
2022-11-23T03:00:01.3769401Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013020.xml (deflated 42%)
2022-11-23T03:00:01.3771051Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013023.xml (deflated 42%)
2022-11-23T03:00:01.3772684Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013029.xml (deflated 43%)
2022-11-23T03:00:01.3774307Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013032.xml (deflated 42%)
2022-11-23T03:00:01.3776033Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013034.xml (deflated 41%)
2022-11-23T03:00:01.3777654Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml (deflated 41%)
2022-11-23T03:00:01.3779507Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013053.xml (deflated 42%)
2022-11-23T03:00:01.3781136Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013056.xml (deflated 41%)
2022-11-23T03:00:01.3782744Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013105.xml (deflated 42%)
2022-11-23T03:00:01.3784369Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013107.xml (deflated 42%)
2022-11-23T03:00:01.3785978Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013114.xml (deflated 41%)
2022-11-23T03:00:01.3787596Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013124.xml (deflated 42%)
2022-11-23T03:00:01.3789336Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013131.xml (deflated 43%)
2022-11-23T03:00:01.3790975Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013133.xml (deflated 42%)
2022-11-23T03:00:01.3792631Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013135.xml (deflated 41%)
2022-11-23T03:00:01.3794277Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013146.xml (deflated 42%)
2022-11-23T03:00:01.3795882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013148.xml (deflated 42%)
2022-11-23T03:00:01.3797514Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013155.xml (deflated 42%)
2022-11-23T03:00:01.3799140Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013157.xml (deflated 42%)
2022-11-23T03:00:01.3800794Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013204.xml (deflated 42%)
2022-11-23T03:00:01.3802417Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013206.xml (deflated 43%)
2022-11-23T03:00:01.3804051Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013209.xml (deflated 43%)
2022-11-23T03:00:01.3805671Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013211.xml (deflated 41%)
2022-11-23T03:00:01.3807278Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013214.xml (deflated 41%)
2022-11-23T03:00:01.3808888Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013216.xml (deflated 41%)
2022-11-23T03:00:01.3810534Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013223.xml (deflated 41%)
2022-11-23T03:00:01.3812194Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013231.xml (deflated 41%)
2022-11-23T03:00:01.3813825Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013238.xml (deflated 41%)
2022-11-23T03:00:01.3815447Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013246.xml (deflated 41%)
2022-11-23T03:00:01.3817168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013253.xml (deflated 41%)
2022-11-23T03:00:01.3819011Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013302.xml (deflated 41%)
2022-11-23T03:00:01.3820725Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013315.xml (deflated 41%)
2022-11-23T03:00:01.3822327Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013327.xml (deflated 41%)
2022-11-23T03:00:01.3823943Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013330.xml (deflated 42%)
2022-11-23T03:00:01.3825536Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013334.xml (deflated 42%)
2022-11-23T03:00:01.3827151Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013336.xml (deflated 42%)
2022-11-23T03:00:01.3828795Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013339.xml (deflated 43%)
2022-11-23T03:00:01.3830586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013341.xml (deflated 42%)
2022-11-23T03:00:01.3832206Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013345.xml (deflated 42%)
2022-11-23T03:00:01.3833811Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013350.xml (deflated 42%)
2022-11-23T03:00:01.3835435Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013354.xml (deflated 40%)
2022-11-23T03:00:01.3837086Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013403.xml (deflated 40%)
2022-11-23T03:00:01.3838720Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013411.xml (deflated 42%)
2022-11-23T03:00:01.3840356Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013416.xml (deflated 42%)
2022-11-23T03:00:01.3841974Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013420.xml (deflated 40%)
2022-11-23T03:00:01.3843603Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013429.xml (deflated 40%)
2022-11-23T03:00:01.3845215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013439.xml (deflated 40%)
2022-11-23T03:00:01.3846883Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013449.xml (deflated 40%)
2022-11-23T03:00:01.3848536Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013458.xml (deflated 42%)
2022-11-23T03:00:01.3850174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013502.xml (deflated 42%)
2022-11-23T03:00:01.3851773Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013504.xml (deflated 42%)
2022-11-23T03:00:01.3853406Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013509.xml (deflated 41%)
2022-11-23T03:00:01.3855055Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013518.xml (deflated 42%)
2022-11-23T03:00:01.3856710Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013522.xml (deflated 42%)
2022-11-23T03:00:01.3858670Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013527.xml (deflated 40%)
2022-11-23T03:00:01.3860287Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013537.xml (deflated 40%)
2022-11-23T03:00:01.3861925Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013547.xml (deflated 42%)
2022-11-23T03:00:01.3863545Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013552.xml (deflated 40%)
2022-11-23T03:00:01.3865159Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013602.xml (deflated 42%)
2022-11-23T03:00:01.3866813Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013604.xml (deflated 41%)
2022-11-23T03:00:01.3868473Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013606.xml (deflated 41%)
2022-11-23T03:00:01.3870203Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013609.xml (deflated 41%)
2022-11-23T03:00:01.3871802Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013611.xml (deflated 41%)
2022-11-23T03:00:01.3873429Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013614.xml (deflated 41%)
2022-11-23T03:00:01.3875048Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013616.xml (deflated 41%)
2022-11-23T03:00:01.3876657Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013618.xml (deflated 41%)
2022-11-23T03:00:01.3878263Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013621.xml (deflated 41%)
2022-11-23T03:00:01.3879939Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013623.xml (deflated 41%)
2022-11-23T03:00:01.3881572Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013626.xml (deflated 41%)
2022-11-23T03:00:01.3883217Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013628.xml (deflated 42%)
2022-11-23T03:00:01.3884859Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013630.xml (deflated 42%)
2022-11-23T03:00:01.3886461Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013634.xml (deflated 41%)
2022-11-23T03:00:01.3888083Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013644.xml (deflated 40%)
2022-11-23T03:00:01.3889703Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013654.xml (deflated 41%)
2022-11-23T03:00:01.3891328Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013656.xml (deflated 41%)
2022-11-23T03:00:01.3892950Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013706.xml (deflated 40%)
2022-11-23T03:00:01.3894610Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013714.xml (deflated 40%)
2022-11-23T03:00:01.3896249Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013734.xml (deflated 41%)
2022-11-23T03:00:01.3898223Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013744.xml (deflated 41%)
2022-11-23T03:00:01.3899848Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013754.xml (deflated 41%)
2022-11-23T03:00:01.3901491Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013804.xml (deflated 41%)
2022-11-23T03:00:01.3903149Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013814.xml (deflated 42%)
2022-11-23T03:00:01.3904795Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013818.xml (deflated 42%)
2022-11-23T03:00:01.3906420Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013822.xml (deflated 42%)
2022-11-23T03:00:01.3908044Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013832.xml (deflated 40%)
2022-11-23T03:00:01.3909674Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013841.xml (deflated 42%)
2022-11-23T03:00:01.3911455Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013846.xml (deflated 40%)
2022-11-23T03:00:01.3913084Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013854.xml (deflated 42%)
2022-11-23T03:00:01.3914705Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013859.xml (deflated 41%)
2022-11-23T03:00:01.3916328Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013908.xml (deflated 41%)
2022-11-23T03:00:01.3917978Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013918.xml (deflated 40%)
2022-11-23T03:00:01.3919612Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013927.xml (deflated 42%)
2022-11-23T03:00:01.3921262Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013932.xml (deflated 42%)
2022-11-23T03:00:01.3922895Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013936.xml (deflated 42%)
2022-11-23T03:00:01.3924525Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013940.xml (deflated 41%)
2022-11-23T03:00:01.3926110Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013950.xml (deflated 41%)
2022-11-23T03:00:01.3927737Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013959.xml (deflated 40%)
2022-11-23T03:00:01.3929374Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014006.xml (deflated 41%)
2022-11-23T03:00:01.3931032Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014013.xml (deflated 42%)
2022-11-23T03:00:01.3932732Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014017.xml (deflated 42%)
2022-11-23T03:00:01.3934390Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014022.xml (deflated 40%)
2022-11-23T03:00:01.3936014Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014028.xml (deflated 41%)
2022-11-23T03:00:01.3937647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014031.xml (deflated 42%)
2022-11-23T03:00:01.3939570Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014033.xml (deflated 41%)
2022-11-23T03:00:01.3941226Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014044.xml (deflated 41%)
2022-11-23T03:00:01.3942849Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014046.xml (deflated 42%)
2022-11-23T03:00:01.3944462Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014048.xml (deflated 40%)
2022-11-23T03:00:01.3946084Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014058.xml (deflated 42%)
2022-11-23T03:00:01.3947736Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014102.xml (deflated 41%)
2022-11-23T03:00:01.3949384Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014109.xml (deflated 40%)
2022-11-23T03:00:01.3951001Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014119.xml (deflated 41%)
2022-11-23T03:00:01.3952746Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014126.xml (deflated 40%)
2022-11-23T03:00:01.3954404Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014133.xml (deflated 40%)
2022-11-23T03:00:01.3956062Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014139.xml (deflated 40%)
2022-11-23T03:00:01.3957727Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014148.xml (deflated 42%)
2022-11-23T03:00:01.3959347Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014151.xml (deflated 43%)
2022-11-23T03:00:01.3960969Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014153.xml (deflated 42%)
2022-11-23T03:00:01.3962586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014155.xml (deflated 42%)
2022-11-23T03:00:01.3964226Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014158.xml (deflated 41%)
2022-11-23T03:00:01.3965876Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014223.xml (deflated 41%)
2022-11-23T03:00:01.3967515Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014248.xml (deflated 42%)
2022-11-23T03:00:01.3969170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014250.xml (deflated 41%)
2022-11-23T03:00:01.3970824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014252.xml (deflated 41%)
2022-11-23T03:00:01.3972490Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014255.xml (deflated 41%)
2022-11-23T03:00:01.3974114Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014257.xml (deflated 41%)
2022-11-23T03:00:01.3975763Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014300.xml (deflated 41%)
2022-11-23T03:00:01.3977430Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014308.xml (deflated 41%)
2022-11-23T03:00:01.3979306Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014316.xml (deflated 41%)
2022-11-23T03:00:01.3981121Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014326.xml (deflated 41%)
2022-11-23T03:00:01.3982804Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014334.xml (deflated 42%)
2022-11-23T03:00:01.3984469Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014338.xml (deflated 42%)
2022-11-23T03:00:01.3986105Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014341.xml (deflated 42%)
2022-11-23T03:00:01.3987698Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014343.xml (deflated 41%)
2022-11-23T03:00:01.3989305Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014345.xml (deflated 40%)
2022-11-23T03:00:01.3990963Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014352.xml (deflated 40%)
2022-11-23T03:00:01.3992745Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014359.xml (deflated 41%)
2022-11-23T03:00:01.3994367Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014402.xml (deflated 42%)
2022-11-23T03:00:01.3996019Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014404.xml (deflated 42%)
2022-11-23T03:00:01.3997662Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014408.xml (deflated 40%)
2022-11-23T03:00:01.3999302Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014418.xml (deflated 41%)
2022-11-23T03:00:01.4001056Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014428.xml (deflated 40%)
2022-11-23T03:00:01.4002672Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014437.xml (deflated 42%)
2022-11-23T03:00:01.4004276Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014441.xml (deflated 41%)
2022-11-23T03:00:01.4005890Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014445.xml (deflated 40%)
2022-11-23T03:00:01.4007488Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014452.xml (deflated 40%)
2022-11-23T03:00:01.4009100Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014459.xml (deflated 42%)
2022-11-23T03:00:01.4010684Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014503.xml (deflated 41%)
2022-11-23T03:00:01.4012317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014506.xml (deflated 41%)
2022-11-23T03:00:01.4013978Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014508.xml (deflated 41%)
2022-11-23T03:00:01.4015627Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014511.xml (deflated 41%)
2022-11-23T03:00:01.4017229Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014513.xml (deflated 41%)
2022-11-23T03:00:01.4019041Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014515.xml (deflated 41%)
2022-11-23T03:00:01.4020787Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014518.xml (deflated 41%)
2022-11-23T03:00:01.4022433Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014520.xml (deflated 41%)
2022-11-23T03:00:01.4024080Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014522.xml (deflated 41%)
2022-11-23T03:00:01.4025736Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014525.xml (deflated 42%)
2022-11-23T03:00:01.4027360Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014527.xml (deflated 40%)
2022-11-23T03:00:01.4028976Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014536.xml (deflated 41%)
2022-11-23T03:00:01.4030579Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014539.xml (deflated 40%)
2022-11-23T03:00:01.4032174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014549.xml (deflated 40%)
2022-11-23T03:00:01.4033930Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014559.xml (deflated 41%)
2022-11-23T03:00:01.4035585Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014601.xml (deflated 41%)
2022-11-23T03:00:01.4037253Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014610.xml (deflated 40%)
2022-11-23T03:00:01.4038859Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014619.xml (deflated 41%)
2022-11-23T03:00:01.4040514Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014621.xml (deflated 41%)
2022-11-23T03:00:01.4042169Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014624.xml (deflated 41%)
2022-11-23T03:00:01.4043824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014626.xml (deflated 41%)
2022-11-23T03:00:01.4045435Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014629.xml (deflated 41%)
2022-11-23T03:00:01.4047055Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014639.xml (deflated 40%)
2022-11-23T03:00:01.4048659Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014649.xml (deflated 41%)
2022-11-23T03:00:01.4050263Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014652.xml (deflated 42%)
2022-11-23T03:00:01.4051877Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014654.xml (deflated 41%)
2022-11-23T03:00:01.4053530Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014657.xml (deflated 43%)
2022-11-23T03:00:01.4055165Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014659.xml (deflated 42%)
2022-11-23T03:00:01.4056749Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014701.xml (deflated 42%)
2022-11-23T03:00:01.4058558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014704.xml (deflated 42%)
2022-11-23T03:00:01.4060159Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014706.xml (deflated 43%)
2022-11-23T03:00:01.4061905Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014709.xml (deflated 40%)
2022-11-23T03:00:01.4063564Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014717.xml (deflated 40%)
2022-11-23T03:00:01.4065199Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014726.xml (deflated 40%)
2022-11-23T03:00:01.4066872Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014735.xml (deflated 42%)
2022-11-23T03:00:01.4068496Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014737.xml (deflated 43%)
2022-11-23T03:00:01.4070100Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014740.xml (deflated 42%)
2022-11-23T03:00:01.4071679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014742.xml (deflated 43%)
2022-11-23T03:00:01.4073289Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014745.xml (deflated 42%)
2022-11-23T03:00:01.4075065Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014747.xml (deflated 42%)
2022-11-23T03:00:01.4076707Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014749.xml (deflated 41%)
2022-11-23T03:00:01.4078283Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014759.xml (deflated 41%)
2022-11-23T03:00:01.4079968Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014802.xml (deflated 41%)
2022-11-23T03:00:01.4081616Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014811.xml (deflated 41%)
2022-11-23T03:00:01.4083262Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014821.xml (deflated 40%)
2022-11-23T03:00:01.4084874Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014835.xml (deflated 40%)
2022-11-23T03:00:01.4086510Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014851.xml (deflated 41%)
2022-11-23T03:00:01.4088158Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014901.xml (deflated 42%)
2022-11-23T03:00:01.4089784Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014903.xml (deflated 42%)
2022-11-23T03:00:01.4091423Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014910.xml (deflated 43%)
2022-11-23T03:00:01.4093084Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014914.xml (deflated 42%)
2022-11-23T03:00:01.4094723Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014917.xml (deflated 42%)
2022-11-23T03:00:01.4096340Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014919.xml (deflated 40%)
2022-11-23T03:00:01.4098082Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014930.xml (deflated 40%)
2022-11-23T03:00:01.4099729Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014939.xml (deflated 40%)
2022-11-23T03:00:01.4101356Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014949.xml (deflated 39%)
2022-11-23T03:00:01.4103135Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014959.xml (deflated 39%)
2022-11-23T03:00:01.4104760Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015009.xml (deflated 40%)
2022-11-23T03:00:01.4106357Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015019.xml (deflated 40%)
2022-11-23T03:00:01.4107976Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015028.xml (deflated 42%)
2022-11-23T03:00:01.4109598Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015032.xml (deflated 41%)
2022-11-23T03:00:01.4111243Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015039.xml (deflated 42%)
2022-11-23T03:00:01.4155985Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015043.xml (deflated 42%)
2022-11-23T03:00:01.4157861Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015053.xml (deflated 42%)
2022-11-23T03:00:01.4159409Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015055.xml (deflated 46%)
2022-11-23T03:00:01.4161006Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015105.xml (deflated 47%)
2022-11-23T03:00:01.4162657Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015115.xml (deflated 48%)
2022-11-23T03:00:01.4164286Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015125.xml (deflated 45%)
2022-11-23T03:00:01.4165942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015135.xml (deflated 41%)
2022-11-23T03:00:01.4167614Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015137.xml (deflated 43%)
2022-11-23T03:00:01.4169309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015140.xml (deflated 43%)
2022-11-23T03:00:01.4170966Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015142.xml (deflated 43%)
2022-11-23T03:00:01.4172584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015144.xml (deflated 44%)
2022-11-23T03:00:01.4174238Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015147.xml (deflated 43%)
2022-11-23T03:00:01.4175928Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015149.xml (deflated 41%)
2022-11-23T03:00:01.4177597Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015152.xml (deflated 40%)
2022-11-23T03:00:01.4179606Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015202.xml (deflated 40%)
2022-11-23T03:00:01.4181238Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015212.xml (deflated 41%)
2022-11-23T03:00:01.4182869Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015215.xml (deflated 41%)
2022-11-23T03:00:01.4184503Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015217.xml (deflated 40%)
2022-11-23T03:00:01.4186279Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015227.xml (deflated 40%)
2022-11-23T03:00:01.4187912Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015238.xml (deflated 42%)
2022-11-23T03:00:01.4189559Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015242.xml (deflated 42%)
2022-11-23T03:00:01.4191197Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015246.xml (deflated 40%)
2022-11-23T03:00:01.4192815Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015255.xml (deflated 40%)
2022-11-23T03:00:01.4194404Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015306.xml (deflated 40%)
2022-11-23T03:00:01.4196024Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015316.xml (deflated 41%)
2022-11-23T03:00:01.4197642Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015318.xml (deflated 41%)
2022-11-23T03:00:01.4199391Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015321.xml (deflated 41%)
2022-11-23T03:00:01.4200989Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015323.xml (deflated 41%)
2022-11-23T03:00:01.4202595Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015325.xml (deflated 41%)
2022-11-23T03:00:01.4204207Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015328.xml (deflated 41%)
2022-11-23T03:00:01.4205827Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015330.xml (deflated 41%)
2022-11-23T03:00:01.4207443Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015333.xml (deflated 41%)
2022-11-23T03:00:01.4209083Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015335.xml (deflated 41%)
2022-11-23T03:00:01.4210753Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015337.xml (deflated 42%)
2022-11-23T03:00:01.4212432Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015340.xml (deflated 41%)
2022-11-23T03:00:01.4214058Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015342.xml (deflated 41%)
2022-11-23T03:00:01.4215689Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015345.xml (deflated 41%)
2022-11-23T03:00:01.4217344Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015347.xml (deflated 41%)
2022-11-23T03:00:01.4219215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015349.xml (deflated 41%)
2022-11-23T03:00:01.4220813Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015352.xml (deflated 41%)
2022-11-23T03:00:01.4222408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015354.xml (deflated 41%)
2022-11-23T03:00:01.4224031Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015357.xml (deflated 41%)
2022-11-23T03:00:01.4225672Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015359.xml (deflated 41%)
2022-11-23T03:00:01.4227454Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015401.xml (deflated 41%)
2022-11-23T03:00:01.4229078Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015404.xml (deflated 41%)
2022-11-23T03:00:01.4230681Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015406.xml (deflated 42%)
2022-11-23T03:00:01.4232369Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015409.xml (deflated 41%)
2022-11-23T03:00:01.4233985Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015411.xml (deflated 41%)
2022-11-23T03:00:01.4235631Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015413.xml (deflated 43%)
2022-11-23T03:00:01.4237250Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015416.xml (deflated 42%)
2022-11-23T03:00:01.4238868Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015418.xml (deflated 41%)
2022-11-23T03:00:01.4240610Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015420.xml (deflated 41%)
2022-11-23T03:00:01.4242268Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015430.xml (deflated 41%)
2022-11-23T03:00:01.4243896Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015432.xml (deflated 41%)
2022-11-23T03:00:01.4245531Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015435.xml (deflated 41%)
2022-11-23T03:00:01.4247119Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015437.xml (deflated 40%)
2022-11-23T03:00:01.4248741Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015446.xml (deflated 40%)
2022-11-23T03:00:01.4250387Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015455.xml (deflated 41%)
2022-11-23T03:00:01.4251995Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015504.xml (deflated 42%)
2022-11-23T03:00:01.4253591Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015506.xml (deflated 42%)
2022-11-23T03:00:01.4255218Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015508.xml (deflated 41%)
2022-11-23T03:00:01.4256856Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015518.xml (deflated 41%)
2022-11-23T03:00:01.4258664Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015527.xml (deflated 42%)
2022-11-23T03:00:01.4260213Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015530.xml (deflated 41%)
2022-11-23T03:00:01.4261756Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015539.xml (deflated 42%)
2022-11-23T03:00:01.4263223Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015542.xml (deflated 42%)
2022-11-23T03:00:01.4264783Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015548.xml (deflated 42%)
2022-11-23T03:00:01.4266243Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015551.xml (deflated 42%)
2022-11-23T03:00:01.4267588Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015553.xml (deflated 43%)
2022-11-23T03:00:01.4268404Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015557.xml (deflated 41%)
2022-11-23T03:00:01.4269230Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015608.xml (deflated 42%)
2022-11-23T03:00:01.4270025Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015610.xml (deflated 42%)
2022-11-23T03:00:01.4270789Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015614.xml (deflated 42%)
2022-11-23T03:00:01.4271559Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015617.xml (deflated 42%)
2022-11-23T03:00:01.4272334Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015624.xml (deflated 42%)
2022-11-23T03:00:01.4273185Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015626.xml (deflated 42%)
2022-11-23T03:00:01.4273937Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015629.xml (deflated 41%)
2022-11-23T03:00:01.4274712Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015638.xml (deflated 41%)
2022-11-23T03:00:01.4275483Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015647.xml (deflated 42%)
2022-11-23T03:00:01.4276253Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015650.xml (deflated 40%)
2022-11-23T03:00:01.4277011Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015659.xml (deflated 42%)
2022-11-23T03:00:01.4277798Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015702.xml (deflated 42%)
2022-11-23T03:00:01.4278569Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015709.xml (deflated 41%)
2022-11-23T03:00:01.4279349Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015718.xml (deflated 42%)
2022-11-23T03:00:01.4280156Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015725.xml (deflated 42%)
2022-11-23T03:00:01.4280933Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015727.xml (deflated 42%)
2022-11-23T03:00:01.4281708Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015730.xml (deflated 41%)
2022-11-23T03:00:01.4282479Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015740.xml (deflated 42%)
2022-11-23T03:00:01.4283254Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015742.xml (deflated 42%)
2022-11-23T03:00:01.4284012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015749.xml (deflated 42%)
2022-11-23T03:00:01.4284781Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015751.xml (deflated 42%)
2022-11-23T03:00:01.4285547Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015758.xml (deflated 42%)
2022-11-23T03:00:01.4286402Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015801.xml (deflated 43%)
2022-11-23T03:00:01.4287167Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015803.xml (deflated 43%)
2022-11-23T03:00:01.4287938Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015805.xml (deflated 40%)
2022-11-23T03:00:01.4288711Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015808.xml (deflated 41%)
2022-11-23T03:00:01.4289479Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015810.xml (deflated 41%)
2022-11-23T03:00:01.4290231Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015817.xml (deflated 41%)
2022-11-23T03:00:01.4290988Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015826.xml (deflated 41%)
2022-11-23T03:00:01.4291737Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015832.xml (deflated 41%)
2022-11-23T03:00:01.4292551Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015841.xml (deflated 40%)
2022-11-23T03:00:01.4293300Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015847.xml (deflated 40%)
2022-11-23T03:00:01.4294068Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015856.xml (deflated 41%)
2022-11-23T03:00:01.4294839Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015905.xml (deflated 41%)
2022-11-23T03:00:01.4295605Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015912.xml (deflated 41%)
2022-11-23T03:00:01.4296355Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015914.xml (deflated 42%)
2022-11-23T03:00:01.4297130Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015918.xml (deflated 41%)
2022-11-23T03:00:01.4298165Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015920.xml (deflated 42%)
2022-11-23T03:00:01.4299106Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015923.xml (deflated 43%)
2022-11-23T03:00:01.4299861Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015925.xml (deflated 42%)
2022-11-23T03:00:01.4300633Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015929.xml (deflated 42%)
2022-11-23T03:00:01.4301403Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015933.xml (deflated 42%)
2022-11-23T03:00:01.4302177Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015938.xml (deflated 40%)
2022-11-23T03:00:01.4302930Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015946.xml (deflated 40%)
2022-11-23T03:00:01.4303687Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015955.xml (deflated 42%)
2022-11-23T03:00:01.4304441Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015959.xml (deflated 42%)
2022-11-23T03:00:01.4305196Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020004.xml (deflated 40%)
2022-11-23T03:00:01.4306029Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020013.xml (deflated 40%)
2022-11-23T03:00:01.4306813Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020022.xml (deflated 41%)
2022-11-23T03:00:01.4307582Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020032.xml (deflated 40%)
2022-11-23T03:00:01.4308351Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020041.xml (deflated 42%)
2022-11-23T03:00:01.4309109Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020046.xml (deflated 43%)
2022-11-23T03:00:01.4309875Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020048.xml (deflated 42%)
2022-11-23T03:00:01.4310647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020052.xml (deflated 41%)
2022-11-23T03:00:01.4311414Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020102.xml (deflated 42%)
2022-11-23T03:00:01.4312236Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020106.xml (deflated 42%)
2022-11-23T03:00:01.4313010Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020111.xml (deflated 40%)
2022-11-23T03:00:01.4313778Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020121.xml (deflated 41%)
2022-11-23T03:00:01.4314545Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020132.xml (deflated 42%)
2022-11-23T03:00:01.4315292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020136.xml (deflated 40%)
2022-11-23T03:00:01.4316061Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020146.xml (deflated 42%)
2022-11-23T03:00:01.4316828Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020148.xml (deflated 42%)
2022-11-23T03:00:01.4317588Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020151.xml (deflated 42%)
2022-11-23T03:00:01.4318337Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020153.xml (deflated 41%)
2022-11-23T03:00:01.4319108Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020155.xml (deflated 41%)
2022-11-23T03:00:01.4319876Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020158.xml (deflated 41%)
2022-11-23T03:00:01.4320645Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020200.xml (deflated 41%)
2022-11-23T03:00:01.4321397Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020202.xml (deflated 42%)
2022-11-23T03:00:01.4322164Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020205.xml (deflated 41%)
2022-11-23T03:00:01.4322928Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020207.xml (deflated 42%)
2022-11-23T03:00:01.4323686Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020210.xml (deflated 42%)
2022-11-23T03:00:01.4324433Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020212.xml (deflated 42%)
2022-11-23T03:00:01.4325254Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020214.xml (deflated 42%)
2022-11-23T03:00:01.4326026Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020219.xml (deflated 41%)
2022-11-23T03:00:01.4326795Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020228.xml (deflated 41%)
2022-11-23T03:00:01.4327569Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020238.xml (deflated 41%)
2022-11-23T03:00:01.4328322Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020240.xml (deflated 41%)
2022-11-23T03:00:01.4329071Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020250.xml (deflated 41%)
2022-11-23T03:00:01.4329837Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020258.xml (deflated 41%)
2022-11-23T03:00:01.4330661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020318.xml (deflated 41%)
2022-11-23T03:00:01.4331414Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020328.xml (deflated 41%)
2022-11-23T03:00:01.4332182Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020338.xml (deflated 41%)
2022-11-23T03:00:01.4332946Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020348.xml (deflated 40%)
2022-11-23T03:00:01.4333704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020358.xml (deflated 42%)
2022-11-23T03:00:01.4334459Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020402.xml (deflated 43%)
2022-11-23T03:00:01.4335219Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020406.xml (deflated 42%)
2022-11-23T03:00:01.4335986Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020416.xml (deflated 41%)
2022-11-23T03:00:01.4336750Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020425.xml (deflated 42%)
2022-11-23T03:00:01.4337503Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020429.xml (deflated 40%)
2022-11-23T03:00:01.4338796Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020438.xml (deflated 42%)
2022-11-23T03:00:01.4339577Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020442.xml (deflated 41%)
2022-11-23T03:00:01.4340343Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020452.xml (deflated 41%)
2022-11-23T03:00:01.4341094Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020502.xml (deflated 40%)
2022-11-23T03:00:01.4341998Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020511.xml (deflated 42%)
2022-11-23T03:00:01.4342761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020515.xml (deflated 42%)
2022-11-23T03:00:01.4343522Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020519.xml (deflated 42%)
2022-11-23T03:00:01.4344349Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020523.xml (deflated 41%)
2022-11-23T03:00:01.4345125Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020533.xml (deflated 41%)
2022-11-23T03:00:01.4345894Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020543.xml (deflated 40%)
2022-11-23T03:00:01.4346654Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020550.xml (deflated 41%)
2022-11-23T03:00:01.4347402Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020557.xml (deflated 42%)
2022-11-23T03:00:01.4348162Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020601.xml (deflated 42%)
2022-11-23T03:00:01.4348921Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020605.xml (deflated 40%)
2022-11-23T03:00:01.4349697Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020612.xml (deflated 41%)
2022-11-23T03:00:01.4350517Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020614.xml (deflated 41%)
2022-11-23T03:00:01.4351287Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020617.xml (deflated 40%)
2022-11-23T03:00:01.4352048Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020627.xml (deflated 41%)
2022-11-23T03:00:01.4352809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020629.xml (deflated 42%)
2022-11-23T03:00:01.4353560Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020632.xml (deflated 40%)
2022-11-23T03:00:01.4354326Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020641.xml (deflated 42%)
2022-11-23T03:00:01.4355102Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020646.xml (deflated 41%)
2022-11-23T03:00:01.4355866Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020652.xml (deflated 40%)
2022-11-23T03:00:01.4356606Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020702.xml (deflated 41%)
2022-11-23T03:00:01.4357378Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020709.xml (deflated 40%)
2022-11-23T03:00:01.4358139Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020716.xml (deflated 40%)
2022-11-23T03:00:01.4358912Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020723.xml (deflated 40%)
2022-11-23T03:00:01.4359662Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020731.xml (deflated 43%)
2022-11-23T03:00:01.4360430Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020734.xml (deflated 43%)
2022-11-23T03:00:01.4361189Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020736.xml (deflated 42%)
2022-11-23T03:00:01.4361951Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020739.xml (deflated 42%)
2022-11-23T03:00:01.4362699Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020741.xml (deflated 41%)
2022-11-23T03:00:01.4363512Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020806.xml (deflated 41%)
2022-11-23T03:00:01.4364296Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020831.xml (deflated 42%)
2022-11-23T03:00:01.4365073Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020833.xml (deflated 42%)
2022-11-23T03:00:01.4365840Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020835.xml (deflated 41%)
2022-11-23T03:00:01.4366589Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020838.xml (deflated 41%)
2022-11-23T03:00:01.4367357Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020840.xml (deflated 41%)
2022-11-23T03:00:01.4368126Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020843.xml (deflated 41%)
2022-11-23T03:00:01.4368895Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020851.xml (deflated 42%)
2022-11-23T03:00:01.4369701Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020859.xml (deflated 41%)
2022-11-23T03:00:01.4370457Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020909.xml (deflated 41%)
2022-11-23T03:00:01.4371230Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020917.xml (deflated 42%)
2022-11-23T03:00:01.4371996Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020921.xml (deflated 42%)
2022-11-23T03:00:01.4372749Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020924.xml (deflated 42%)
2022-11-23T03:00:01.4373514Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020926.xml (deflated 41%)
2022-11-23T03:00:01.4374287Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020929.xml (deflated 40%)
2022-11-23T03:00:01.4375054Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020935.xml (deflated 40%)
2022-11-23T03:00:01.4375801Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020942.xml (deflated 42%)
2022-11-23T03:00:01.4376566Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020944.xml (deflated 42%)
2022-11-23T03:00:01.4377331Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020947.xml (deflated 42%)
2022-11-23T03:00:01.4378441Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020951.xml (deflated 40%)
2022-11-23T03:00:01.4379223Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021001.xml (deflated 41%)
2022-11-23T03:00:01.4380026Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021011.xml (deflated 40%)
2022-11-23T03:00:01.4380802Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021020.xml (deflated 42%)
2022-11-23T03:00:01.4381566Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021024.xml (deflated 41%)
2022-11-23T03:00:01.4382318Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021028.xml (deflated 41%)
2022-11-23T03:00:01.4383172Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021035.xml (deflated 40%)
2022-11-23T03:00:01.4383958Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021042.xml (deflated 42%)
2022-11-23T03:00:01.4384732Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021046.xml (deflated 42%)
2022-11-23T03:00:01.4385485Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021048.xml (deflated 42%)
2022-11-23T03:00:01.4386250Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021051.xml (deflated 42%)
2022-11-23T03:00:01.4387019Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021053.xml (deflated 42%)
2022-11-23T03:00:01.4387794Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021056.xml (deflated 41%)
2022-11-23T03:00:01.4388611Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021058.xml (deflated 41%)
2022-11-23T03:00:01.4389378Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021100.xml (deflated 42%)
2022-11-23T03:00:01.4390146Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021103.xml (deflated 42%)
2022-11-23T03:00:01.4390915Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021105.xml (deflated 41%)
2022-11-23T03:00:01.4391663Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021108.xml (deflated 42%)
2022-11-23T03:00:01.4392432Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021110.xml (deflated 41%)
2022-11-23T03:00:01.4393193Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021119.xml (deflated 42%)
2022-11-23T03:00:01.4393968Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021121.xml (deflated 41%)
2022-11-23T03:00:01.4394715Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021132.xml (deflated 40%)
2022-11-23T03:00:01.4395487Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021141.xml (deflated 41%)
2022-11-23T03:00:01.4396256Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021144.xml (deflated 40%)
2022-11-23T03:00:01.4397028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021152.xml (deflated 41%)
2022-11-23T03:00:01.4397775Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021201.xml (deflated 42%)
2022-11-23T03:00:01.4398540Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021204.xml (deflated 42%)
2022-11-23T03:00:01.4399307Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021206.xml (deflated 42%)
2022-11-23T03:00:01.4400073Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021209.xml (deflated 42%)
2022-11-23T03:00:01.4400824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021211.xml (deflated 41%)
2022-11-23T03:00:01.4401642Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021221.xml (deflated 41%)
2022-11-23T03:00:01.4402420Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021232.xml (deflated 42%)
2022-11-23T03:00:01.4403191Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021234.xml (deflated 42%)
2022-11-23T03:00:01.4403941Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021237.xml (deflated 41%)
2022-11-23T03:00:01.4404709Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021239.xml (deflated 43%)
2022-11-23T03:00:01.4405474Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021241.xml (deflated 42%)
2022-11-23T03:00:01.4406242Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021244.xml (deflated 42%)
2022-11-23T03:00:01.4406997Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021246.xml (deflated 42%)
2022-11-23T03:00:01.4407818Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021249.xml (deflated 43%)
2022-11-23T03:00:01.4408586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021251.xml (deflated 40%)
2022-11-23T03:00:01.4409358Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021259.xml (deflated 40%)
2022-11-23T03:00:01.4410107Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021308.xml (deflated 40%)
2022-11-23T03:00:01.4410875Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021317.xml (deflated 43%)
2022-11-23T03:00:01.4411647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021320.xml (deflated 43%)
2022-11-23T03:00:01.4412418Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021322.xml (deflated 43%)
2022-11-23T03:00:01.4413177Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021324.xml (deflated 43%)
2022-11-23T03:00:01.4413927Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021327.xml (deflated 42%)
2022-11-23T03:00:01.4414696Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021329.xml (deflated 42%)
2022-11-23T03:00:01.4415464Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021331.xml (deflated 42%)
2022-11-23T03:00:01.4416228Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021341.xml (deflated 41%)
2022-11-23T03:00:01.4416988Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021344.xml (deflated 40%)
2022-11-23T03:00:01.4417758Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021353.xml (deflated 41%)
2022-11-23T03:00:01.4418885Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021403.xml (deflated 40%)
2022-11-23T03:00:01.4419661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021417.xml (deflated 40%)
2022-11-23T03:00:01.4420409Z   adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20221123021513.xml (deflated 42%)
2022-11-23T03:00:01.4421241Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20221123021517.xml (deflated 71%)
2022-11-23T03:00:01.4422004Z   adding: test/test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20221123021521.xml (deflated 62%)
2022-11-23T03:00:01.4422803Z   adding: test/test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20221123021525.xml (deflated 71%)
2022-11-23T03:00:01.4423615Z   adding: test/test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20221123021525.xml (deflated 40%)
2022-11-23T03:00:01.4424405Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221123021529.xml (deflated 62%)
2022-11-23T03:00:01.4425136Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221123021529.xml (deflated 69%)
2022-11-23T03:00:01.4425871Z   adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20221123021535.xml (deflated 40%)
2022-11-23T03:00:01.4426598Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123021542.xml (deflated 41%)
2022-11-23T03:00:01.4427423Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123021550.xml (deflated 45%)
2022-11-23T03:00:01.4428203Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123021558.xml (deflated 41%)
2022-11-23T03:00:01.4429027Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221123021606.xml (deflated 60%)
2022-11-23T03:00:01.4429870Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20221123021615.xml (deflated 60%)
2022-11-23T03:00:01.4430670Z   adding: test/test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20221123021624.xml (deflated 54%)
2022-11-23T03:00:01.4431464Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221123021632.xml (deflated 68%)
2022-11-23T03:00:01.4432326Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123021644.xml (deflated 59%)
2022-11-23T03:00:01.4433186Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221123021656.xml (deflated 73%)
2022-11-23T03:00:01.4434017Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221123021709.xml (deflated 76%)
2022-11-23T03:00:01.4434881Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221123021725.xml (deflated 68%)
2022-11-23T03:00:01.4435829Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221123021725.xml (deflated 43%)
2022-11-23T03:00:01.4436869Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123021725.xml (deflated 44%)
2022-11-23T03:00:01.4437729Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123021741.xml (deflated 61%)
2022-11-23T03:00:01.4438473Z   adding: test/test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20221123021758.xml (deflated 78%)
2022-11-23T03:00:01.4439276Z   adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_tp_style/TEST-TensorParallelStyleTest-20221123021817.xml (deflated 82%)
2022-11-23T03:00:01.4440193Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123021840.xml (deflated 67%)
2022-11-23T03:00:01.4441107Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123021840.xml (deflated 43%)
2022-11-23T03:00:01.4442133Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123021840.xml (deflated 45%)
2022-11-23T03:00:01.4443103Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123021905.xml (deflated 56%)
2022-11-23T03:00:01.4443956Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123021905.xml (deflated 77%)
2022-11-23T03:00:01.4444725Z   adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20221123021932.xml (deflated 76%)
2022-11-23T03:00:01.4445521Z   adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20221123021932.xml (deflated 42%)
2022-11-23T03:00:01.4446350Z   adding: test/test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20221123022003.xml (deflated 84%)
2022-11-23T03:00:01.4447056Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221123022035.xml (deflated 38%)
2022-11-23T03:00:01.4447788Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123022042.xml (deflated 41%)
2022-11-23T03:00:01.4448593Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123022046.xml (deflated 40%)
2022-11-23T03:00:01.4449394Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123022049.xml (deflated 41%)
2022-11-23T03:00:01.4450192Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123022053.xml (deflated 42%)
2022-11-23T03:00:01.4451021Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123022057.xml (deflated 42%)
2022-11-23T03:00:01.4451847Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123022104.xml (deflated 42%)
2022-11-23T03:00:01.4452681Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123022113.xml (deflated 42%)
2022-11-23T03:00:01.4453515Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123022119.xml (deflated 42%)
2022-11-23T03:00:01.4454270Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123022128.xml (deflated 39%)
2022-11-23T03:00:01.4454946Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123022132.xml (deflated 39%)
2022-11-23T03:00:01.4455641Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123022136.xml (deflated 39%)
2022-11-23T03:00:01.4456408Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123022140.xml (deflated 63%)
2022-11-23T03:00:01.4457292Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123022140.xml (deflated 84%)
2022-11-23T03:00:01.4458541Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221123022220.xml (deflated 91%)
2022-11-23T03:00:01.4459374Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221123022302.xml (deflated 84%)
2022-11-23T03:00:01.4460196Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022351.xml (deflated 40%)
2022-11-23T03:00:01.4460899Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022354.xml (deflated 39%)
2022-11-23T03:00:01.4461563Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022358.xml (deflated 39%)
2022-11-23T03:00:01.4462240Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022402.xml (deflated 39%)
2022-11-23T03:00:01.4462910Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022406.xml (deflated 39%)
2022-11-23T03:00:01.4463581Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022410.xml (deflated 39%)
2022-11-23T03:00:01.4464272Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022414.xml (deflated 40%)
2022-11-23T03:00:01.4465001Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022418.xml (deflated 40%)
2022-11-23T03:00:01.4465782Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123022422.xml (deflated 40%)
2022-11-23T03:00:01.4466501Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022424.xml (deflated 40%)
2022-11-23T03:00:01.4467205Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022428.xml (deflated 39%)
2022-11-23T03:00:01.4467914Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123022432.xml (deflated 39%)
2022-11-23T03:00:01.4468612Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123022436.xml (deflated 39%)
2022-11-23T03:00:01.4469321Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022440.xml (deflated 40%)
2022-11-23T03:00:01.4470016Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022444.xml (deflated 39%)
2022-11-23T03:00:01.4470729Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022447.xml (deflated 39%)
2022-11-23T03:00:01.4471429Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022451.xml (deflated 39%)
2022-11-23T03:00:01.4472110Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022455.xml (deflated 39%)
2022-11-23T03:00:01.4472806Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022459.xml (deflated 40%)
2022-11-23T03:00:01.4473502Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022513.xml (deflated 38%)
2022-11-23T03:00:01.4474186Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022517.xml (deflated 38%)
2022-11-23T03:00:01.4474853Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022521.xml (deflated 39%)
2022-11-23T03:00:01.4475535Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022525.xml (deflated 39%)
2022-11-23T03:00:01.4476205Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022529.xml (deflated 38%)
2022-11-23T03:00:01.4476877Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022533.xml (deflated 38%)
2022-11-23T03:00:01.4477529Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022537.xml (deflated 38%)
2022-11-23T03:00:01.4478202Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022541.xml (deflated 39%)
2022-11-23T03:00:01.4478908Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022545.xml (deflated 38%)
2022-11-23T03:00:01.4479588Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022550.xml (deflated 39%)
2022-11-23T03:00:01.4480337Z   adding: test/test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20221123022555.xml (deflated 84%)
2022-11-23T03:00:01.4481056Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123022711.xml (deflated 81%)
2022-11-23T03:00:01.4481763Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123022711.xml (deflated 89%)
2022-11-23T03:00:01.4482566Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221123022853.xml (deflated 43%)
2022-11-23T03:00:01.4483425Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221123022853.xml (deflated 44%)
2022-11-23T03:00:01.4484263Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221123022853.xml (deflated 60%)
2022-11-23T03:00:01.4485123Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221123022853.xml (deflated 58%)
2022-11-23T03:00:01.4485938Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221123022853.xml (deflated 58%)
2022-11-23T03:00:01.4486751Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221123022853.xml (deflated 61%)
2022-11-23T03:00:01.4487532Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221123022853.xml (deflated 61%)
2022-11-23T03:00:01.4488368Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221123022853.xml (deflated 88%)
2022-11-23T03:00:01.4489241Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221123022853.xml (deflated 69%)
2022-11-23T03:00:01.4490127Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221123022853.xml (deflated 87%)
2022-11-23T03:00:01.4491016Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221123022853.xml (deflated 82%)
2022-11-23T03:00:01.4491937Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221123022853.xml (deflated 62%)
2022-11-23T03:00:01.4492812Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221123023124.xml (deflated 91%)
2022-11-23T03:00:01.4493662Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221123023124.xml (deflated 43%)
2022-11-23T03:00:01.4494468Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123023501.xml (deflated 95%)
2022-11-23T03:00:01.4495177Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024257.xml (deflated 38%)
2022-11-23T03:00:01.4495842Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024306.xml (deflated 38%)
2022-11-23T03:00:01.4496497Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024315.xml (deflated 39%)
2022-11-23T03:00:01.4497136Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024321.xml (deflated 38%)
2022-11-23T03:00:01.4497837Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024329.xml (deflated 38%)
2022-11-23T03:00:01.4498873Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024335.xml (deflated 39%)
2022-11-23T03:00:01.4499531Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024342.xml (deflated 39%)
2022-11-23T03:00:01.4500169Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024348.xml (deflated 39%)
2022-11-23T03:00:01.4500824Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024354.xml (deflated 38%)
2022-11-23T03:00:01.4501474Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024402.xml (deflated 38%)
2022-11-23T03:00:01.4502127Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024410.xml (deflated 38%)
2022-11-23T03:00:01.4502767Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024418.xml (deflated 37%)
2022-11-23T03:00:01.4503502Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024424.xml (deflated 38%)
2022-11-23T03:00:01.4504158Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024433.xml (deflated 38%)
2022-11-23T03:00:01.4504813Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024441.xml (deflated 39%)
2022-11-23T03:00:01.4505444Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024447.xml (deflated 38%)
2022-11-23T03:00:01.4506094Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024455.xml (deflated 38%)
2022-11-23T03:00:01.4506740Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024503.xml (deflated 37%)
2022-11-23T03:00:01.4507395Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024512.xml (deflated 38%)
2022-11-23T03:00:01.4508053Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024520.xml (deflated 38%)
2022-11-23T03:00:01.4508738Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024529.xml (deflated 38%)
2022-11-23T03:00:01.4509422Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024538.xml (deflated 37%)
2022-11-23T03:00:01.4510093Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024547.xml (deflated 38%)
2022-11-23T03:00:01.4510743Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024556.xml (deflated 38%)
2022-11-23T03:00:01.4511433Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024605.xml (deflated 38%)
2022-11-23T03:00:01.4512111Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024614.xml (deflated 38%)
2022-11-23T03:00:01.4512855Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024623.xml (deflated 42%)
2022-11-23T03:00:01.4513636Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024632.xml (deflated 41%)
2022-11-23T03:00:01.4514417Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024642.xml (deflated 41%)
2022-11-23T03:00:01.4515212Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024651.xml (deflated 41%)
2022-11-23T03:00:01.4516001Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024701.xml (deflated 41%)
2022-11-23T03:00:01.4516854Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024710.xml (deflated 42%)
2022-11-23T03:00:01.4517657Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024719.xml (deflated 42%)
2022-11-23T03:00:01.4518442Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024728.xml (deflated 41%)
2022-11-23T03:00:01.4519233Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024737.xml (deflated 41%)
2022-11-23T03:00:01.4520000Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024746.xml (deflated 44%)
2022-11-23T03:00:01.4520777Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024754.xml (deflated 44%)
2022-11-23T03:00:01.4521566Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024802.xml (deflated 43%)
2022-11-23T03:00:01.4522403Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024811.xml (deflated 43%)
2022-11-23T03:00:01.4523171Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024819.xml (deflated 45%)
2022-11-23T03:00:01.4523946Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024828.xml (deflated 45%)
2022-11-23T03:00:01.4524725Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024836.xml (deflated 46%)
2022-11-23T03:00:01.4525511Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024844.xml (deflated 46%)
2022-11-23T03:00:01.4526279Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024853.xml (deflated 44%)
2022-11-23T03:00:01.4527059Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024901.xml (deflated 46%)
2022-11-23T03:00:01.4527839Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024910.xml (deflated 46%)
2022-11-23T03:00:01.4528614Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024918.xml (deflated 44%)
2022-11-23T03:00:01.4529381Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024926.xml (deflated 44%)
2022-11-23T03:00:01.4530157Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024935.xml (deflated 42%)
2022-11-23T03:00:01.4530933Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024944.xml (deflated 41%)
2022-11-23T03:00:01.4531771Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024953.xml (deflated 41%)
2022-11-23T03:00:01.4532540Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025001.xml (deflated 45%)
2022-11-23T03:00:01.4533325Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025010.xml (deflated 44%)
2022-11-23T03:00:01.4534103Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025019.xml (deflated 42%)
2022-11-23T03:00:01.4534881Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025025.xml (deflated 44%)
2022-11-23T03:00:01.4535687Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025035.xml (deflated 41%)
2022-11-23T03:00:01.4536479Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025044.xml (deflated 41%)
2022-11-23T03:00:01.4537266Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025051.xml (deflated 41%)
2022-11-23T03:00:01.4538412Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025100.xml (deflated 41%)
2022-11-23T03:00:01.4539224Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025109.xml (deflated 43%)
2022-11-23T03:00:01.4540013Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025118.xml (deflated 42%)
2022-11-23T03:00:01.4540802Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025122.xml (deflated 42%)
2022-11-23T03:00:01.4541596Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025126.xml (deflated 42%)
2022-11-23T03:00:01.4542470Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025130.xml (deflated 42%)
2022-11-23T03:00:01.4543244Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025134.xml (deflated 42%)
2022-11-23T03:00:01.4544031Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025138.xml (deflated 42%)
2022-11-23T03:00:01.4544814Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025142.xml (deflated 42%)
2022-11-23T03:00:01.4545588Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025151.xml (deflated 41%)
2022-11-23T03:00:01.4546357Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025200.xml (deflated 42%)
2022-11-23T03:00:01.4547139Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025209.xml (deflated 41%)
2022-11-23T03:00:01.4547923Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025218.xml (deflated 41%)
2022-11-23T03:00:01.4548705Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025229.xml (deflated 42%)
2022-11-23T03:00:01.4549473Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025236.xml (deflated 41%)
2022-11-23T03:00:01.4550256Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025242.xml (deflated 42%)
2022-11-23T03:00:01.4551040Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025252.xml (deflated 42%)
2022-11-23T03:00:01.4551826Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025301.xml (deflated 41%)
2022-11-23T03:00:01.4552593Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025310.xml (deflated 42%)
2022-11-23T03:00:01.4553365Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025320.xml (deflated 42%)
2022-11-23T03:00:01.4554145Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025326.xml (deflated 42%)
2022-11-23T03:00:01.4554928Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025332.xml (deflated 41%)
2022-11-23T03:00:01.4555749Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025340.xml (deflated 42%)
2022-11-23T03:00:01.4556540Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025346.xml (deflated 42%)
2022-11-23T03:00:01.4557330Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025356.xml (deflated 42%)
2022-11-23T03:00:01.4558111Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025405.xml (deflated 43%)
2022-11-23T03:00:01.4558883Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025430.xml (deflated 44%)
2022-11-23T03:00:01.4559655Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025439.xml (deflated 42%)
2022-11-23T03:00:01.4560442Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025448.xml (deflated 41%)
2022-11-23T03:00:01.4561270Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025454.xml (deflated 41%)
2022-11-23T03:00:01.4562031Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025503.xml (deflated 41%)
2022-11-23T03:00:01.4562816Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025512.xml (deflated 41%)
2022-11-23T03:00:01.4563600Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025523.xml (deflated 41%)
2022-11-23T03:00:01.4564367Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025532.xml (deflated 41%)
2022-11-23T03:00:01.4565100Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025539.xml (deflated 41%)
2022-11-23T03:00:01.4565853Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025545.xml (deflated 42%)
2022-11-23T03:00:01.4566602Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025547.xml (deflated 42%)
2022-11-23T03:00:01.4567345Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025553.xml (deflated 40%)
2022-11-23T03:00:01.4568069Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025600.xml (deflated 41%)
2022-11-23T03:00:01.4568806Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025606.xml (deflated 41%)
2022-11-23T03:00:01.4569548Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025612.xml (deflated 42%)
2022-11-23T03:00:01.4570286Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025615.xml (deflated 41%)
2022-11-23T03:00:01.4571134Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123025621.xml (deflated 42%)
2022-11-23T03:00:01.4572092Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123025629.xml (deflated 42%)
2022-11-23T03:00:01.4573054Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123025636.xml (deflated 43%)
2022-11-23T03:00:01.4574003Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123025644.xml (deflated 42%)
2022-11-23T03:00:01.4574908Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123025652.xml (deflated 41%)
2022-11-23T03:00:01.4575678Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025654.xml (deflated 38%)
2022-11-23T03:00:01.4576429Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025702.xml (deflated 38%)
2022-11-23T03:00:01.4577170Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025711.xml (deflated 38%)
2022-11-23T03:00:01.4578121Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025720.xml (deflated 39%)
2022-11-23T03:00:01.4578963Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025729.xml (deflated 38%)
2022-11-23T03:00:01.4579738Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025737.xml (deflated 39%)
2022-11-23T03:00:01.4580479Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025746.xml (deflated 38%)
2022-11-23T03:00:01.4581322Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025755.xml (deflated 38%)
2022-11-23T03:00:01.4582032Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025803.xml (deflated 38%)
2022-11-23T03:00:01.4582768Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025812.xml (deflated 39%)
2022-11-23T03:00:01.4583507Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025826.xml (deflated 40%)
2022-11-23T03:00:01.4584244Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025834.xml (deflated 38%)
2022-11-23T03:00:01.4584967Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025843.xml (deflated 39%)
2022-11-23T03:00:01.4585702Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025850.xml (deflated 38%)
2022-11-23T03:00:01.4586444Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025859.xml (deflated 39%)
2022-11-23T03:00:01.4587175Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025908.xml (deflated 38%)
2022-11-23T03:00:01.4587890Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025916.xml (deflated 39%)
2022-11-23T03:00:01.4588624Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025925.xml (deflated 39%)
2022-11-23T03:00:01.4589363Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123025939.xml (deflated 39%)
2022-11-23T03:00:01.4590087Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123025946.xml (deflated 39%)
2022-11-23T03:00:01.4590769Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123025950.xml (deflated 40%)
2022-11-23T03:00:01.4630870Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T03:00:01.4631277Z # Remove any previous test reports if they exist
2022-11-23T03:00:01.4631598Z rm -f usage-log-*.zip
2022-11-23T03:00:01.4631969Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-11-23T03:00:01.4632348Z # so check to see if the file exists first
2022-11-23T03:00:01.4632664Z if [ -f 'usage_log.txt' ]; then
2022-11-23T03:00:01.4633001Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-11-23T03:00:01.4633300Z fi
2022-11-23T03:00:01.4646546Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:00:01.4646865Z env:
2022-11-23T03:00:01.4647114Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:01.4647365Z   GPU_FLAG: --gpus all
2022-11-23T03:00:01.4647729Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:01.4648200Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205
2022-11-23T03:00:01.4648539Z ##[endgroup]
2022-11-23T03:00:01.5583375Z   adding: usage_log.txt (deflated 95%)
2022-11-23T03:00:01.5632272Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T03:00:01.5632548Z with:
2022-11-23T03:00:01.5632825Z   s3-prefix: pytorch/pytorch/3528293562/1/artifact

2022-11-23T03:00:01.5633121Z   retention-days: 14
2022-11-23T03:00:01.5633371Z   if-no-files-found: warn
2022-11-23T03:00:01.5633644Z   path: test-jsons-*.zip
2022-11-23T03:00:01.5633898Z   name: artifact
2022-11-23T03:00:01.5634132Z   s3-bucket: gha-artifacts
2022-11-23T03:00:01.5634403Z   region: us-east-1
2022-11-23T03:00:01.5634634Z env:
2022-11-23T03:00:01.5634853Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:01.5635252Z   GPU_FLAG: --gpus all
2022-11-23T03:00:01.5635610Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:01.5635938Z ##[endgroup]
2022-11-23T03:00:02.0095881Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T03:00:02.0096650Z With the provided path, there will be 1 file uploaded
2022-11-23T03:00:02.0097062Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact
2022-11-23T03:00:02.0108605Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205.zip
2022-11-23T03:00:02.2478726Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205.zip
2022-11-23T03:00:02.2637930Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T03:00:02.2638202Z with:
2022-11-23T03:00:02.2638475Z   s3-prefix: pytorch/pytorch/3528293562/1/artifact

2022-11-23T03:00:02.2638777Z   retention-days: 14
2022-11-23T03:00:02.2639029Z   if-no-files-found: error
2022-11-23T03:00:02.2639315Z   path: test-reports-*.zip
2022-11-23T03:00:02.2639568Z   name: artifact
2022-11-23T03:00:02.2639798Z   s3-bucket: gha-artifacts
2022-11-23T03:00:02.2640054Z   region: us-east-1
2022-11-23T03:00:02.2640281Z env:
2022-11-23T03:00:02.2640498Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:02.2640761Z   GPU_FLAG: --gpus all
2022-11-23T03:00:02.2641117Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:02.2641460Z ##[endgroup]
2022-11-23T03:00:02.7056378Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T03:00:02.7057236Z With the provided path, there will be 1 file uploaded
2022-11-23T03:00:02.7057608Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact
2022-11-23T03:00:02.7068979Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205.zip
2022-11-23T03:00:03.2316839Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205.zip
2022-11-23T03:00:03.2473775Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T03:00:03.2474075Z with:
2022-11-23T03:00:03.2474358Z   s3-prefix: pytorch/pytorch/3528293562/1/artifact

2022-11-23T03:00:03.2474657Z   retention-days: 14
2022-11-23T03:00:03.2474909Z   if-no-files-found: ignore
2022-11-23T03:00:03.2475185Z   path: usage-log-*.zip
2022-11-23T03:00:03.2475441Z   name: artifact
2022-11-23T03:00:03.2475676Z   s3-bucket: gha-artifacts
2022-11-23T03:00:03.2475942Z   region: us-east-1
2022-11-23T03:00:03.2476174Z env:
2022-11-23T03:00:03.2476396Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:03.2476663Z   GPU_FLAG: --gpus all
2022-11-23T03:00:03.2477026Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:03.2477352Z ##[endgroup]
2022-11-23T03:00:03.6847180Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T03:00:03.6848243Z With the provided path, there will be 1 file uploaded
2022-11-23T03:00:03.6848655Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact
2022-11-23T03:00:03.6859195Z Starting upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205.zip
2022-11-23T03:00:04.0889178Z Finished upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655200205.zip
2022-11-23T03:00:04.1061249Z ##[group]Run # shellcheck disable=SC2156
2022-11-23T03:00:04.1061904Z # shellcheck disable=SC2156
2022-11-23T03:00:04.1062715Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2022-11-23T03:00:04.1079690Z shell: /usr/bin/bash -e {0}
2022-11-23T03:00:04.1079934Z env:
2022-11-23T03:00:04.1080180Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:04.1080453Z   GPU_FLAG: --gpus all
2022-11-23T03:00:04.1080799Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:04.1081145Z ##[endgroup]
2022-11-23T03:00:04.4130813Z ##[group]Run set -x
2022-11-23T03:00:04.4131125Z set -x
2022-11-23T03:00:04.4131535Z python3 -m pip install -r requirements.txt
2022-11-23T03:00:04.4131877Z python3 -m pip install boto3==1.19.12
2022-11-23T03:00:04.4132274Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T03:00:04.4145093Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:00:04.4145392Z env:
2022-11-23T03:00:04.4145636Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:04.4145887Z   GPU_FLAG: --gpus all
2022-11-23T03:00:04.4146244Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:04.4146611Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T03:00:04.4146857Z   BRANCH: master
2022-11-23T03:00:04.4147107Z   TEST_CONFIG: distributed
2022-11-23T03:00:04.4147359Z   SHARD_NUMBER: 2
2022-11-23T03:00:04.4147657Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T03:00:04.4148013Z   PR_NUMBER: 
2022-11-23T03:00:04.4148255Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T03:00:04.4148546Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T03:00:04.4148853Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T03:00:04.4149101Z   TAG: 
2022-11-23T03:00:04.4149329Z   WORKFLOW_ID: 3528293562
2022-11-23T03:00:04.4149760Z   GITHUB_TOKEN: ***
2022-11-23T03:00:04.4150008Z   GHA_WORKFLOW_JOB_ID: 9655200205
2022-11-23T03:00:04.4150266Z ##[endgroup]
2022-11-23T03:00:04.4179219Z + python3 -m pip install -r requirements.txt
2022-11-23T03:00:04.7050826Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T03:00:04.7871568Z Collecting astunparse
2022-11-23T03:00:04.8091340Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-11-23T03:00:04.8426148Z Collecting expecttest
2022-11-23T03:00:04.8573698Z   Downloading expecttest-0.1.4-py3-none-any.whl (6.5 kB)
2022-11-23T03:00:04.8973140Z Collecting future
2022-11-23T03:00:04.9015492Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-11-23T03:00:06.8924591Z Collecting hypothesis
2022-11-23T03:00:06.8993108Z   Downloading hypothesis-6.58.0-py3-none-any.whl (396 kB)
2022-11-23T03:00:07.7432555Z Collecting numpy
2022-11-23T03:00:07.7512562Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-11-23T03:00:08.0793479Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-11-23T03:00:08.2024327Z Collecting pyyaml
2022-11-23T03:00:08.2083861Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-11-23T03:00:08.2298206Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-11-23T03:00:08.2480088Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-11-23T03:00:08.3087697Z Collecting six
2022-11-23T03:00:08.3128518Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2022-11-23T03:00:08.3564286Z Collecting types-dataclasses
2022-11-23T03:00:08.3645926Z   Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB)
2022-11-23T03:00:08.4076212Z Collecting typing_extensions
2022-11-23T03:00:08.4116683Z   Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
2022-11-23T03:00:08.4684848Z Collecting sympy
2022-11-23T03:00:08.4783527Z   Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB)
2022-11-23T03:00:08.6689889Z Collecting filelock
2022-11-23T03:00:08.6751616Z   Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
2022-11-23T03:00:08.7692042Z Collecting networkx
2022-11-23T03:00:08.7822836Z   Downloading networkx-2.6.3-py3-none-any.whl (1.9 MB)
2022-11-23T03:00:08.9061505Z Collecting jinja2
2022-11-23T03:00:08.9105044Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2022-11-23T03:00:09.0114567Z Collecting wheel<1.0,>=0.23.0
2022-11-23T03:00:09.0171690Z   Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
2022-11-23T03:00:09.0633514Z Collecting sortedcontainers<3.0.0,>=2.1.0
2022-11-23T03:00:09.0688137Z   Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
2022-11-23T03:00:09.1060868Z Collecting exceptiongroup>=1.0.0; python_version < "3.11"
2022-11-23T03:00:09.1101795Z   Downloading exceptiongroup-1.0.4-py3-none-any.whl (14 kB)
2022-11-23T03:00:09.1605869Z Collecting attrs>=19.2.0
2022-11-23T03:00:09.1649281Z   Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
2022-11-23T03:00:09.2112939Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12)
2022-11-23T03:00:09.2337292Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-11-23T03:00:09.2348960Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-11-23T03:00:09.2364250Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-11-23T03:00:09.2648568Z Collecting mpmath>=0.19
2022-11-23T03:00:09.2749716Z   Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
2022-11-23T03:00:09.4290477Z Collecting MarkupSafe>=2.0
2022-11-23T03:00:09.4349732Z   Downloading MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
2022-11-23T03:00:09.4443591Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-11-23T03:00:09.6408130Z Installing collected packages: wheel, six, astunparse, expecttest, future, sortedcontainers, exceptiongroup, attrs, hypothesis, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy, filelock, networkx, MarkupSafe, jinja2
2022-11-23T03:00:09.6695827Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:00:09.6696498Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:00:09.7146113Z     Running setup.py install for future: started
2022-11-23T03:00:10.3783113Z     Running setup.py install for future: finished with status 'done'
2022-11-23T03:00:10.6888382Z   WARNING: The script hypothesis is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:00:10.6889037Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:00:12.6717757Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:00:12.6718452Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:00:21.6034254Z   WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:00:21.6035357Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:00:22.8561754Z Successfully installed MarkupSafe-2.1.1 astunparse-1.6.3 attrs-22.1.0 exceptiongroup-1.0.4 expecttest-0.1.4 filelock-3.8.0 future-0.18.2 hypothesis-6.58.0 jinja2-3.1.2 mpmath-1.2.1 networkx-2.6.3 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.4.0 wheel-0.38.4
2022-11-23T03:00:22.9329252Z + python3 -m pip install boto3==1.19.12
2022-11-23T03:00:23.2249731Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T03:00:24.2187365Z Collecting boto3==1.19.12
2022-11-23T03:00:24.2422259Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-11-23T03:00:24.2983682Z Collecting jmespath<1.0.0,>=0.7.1
2022-11-23T03:00:24.3028225Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2022-11-23T03:00:24.3548985Z Collecting s3transfer<0.6.0,>=0.5.0
2022-11-23T03:00:24.3595110Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-11-23T03:00:25.5870735Z Collecting botocore<1.23.0,>=1.22.12
2022-11-23T03:00:25.5928126Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-11-23T03:00:25.8352583Z Collecting python-dateutil<3.0.0,>=2.1
2022-11-23T03:00:25.8398179Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2022-11-23T03:00:25.8585260Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12)
2022-11-23T03:00:25.8807505Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-11-23T03:00:26.0950012Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
2022-11-23T03:00:26.9846098Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2022-11-23T03:00:27.0387667Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T03:00:41.2302409Z [scribe] Scribe access token not provided, sending report via boto3...
2022-11-23T03:00:41.2302692Z 
2022-11-23T03:00:41.2303024Z ----- Historic stats comparison result ------
2022-11-23T03:00:41.2305519Z 
2022-11-23T03:00:41.2305912Z     job: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T03:00:41.2306266Z     commit: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T03:00:41.2307501Z 
2022-11-23T03:00:41.2308114Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-11-23T03:00:41.2308373Z 
2022-11-23T03:00:41.2308480Z     : (master)
2022-11-23T03:00:41.2308715Z     |
2022-11-23T03:00:41.2308986Z     * 1cfd3858ac (HEAD)              total time  3095.28s
2022-11-23T03:00:41.2309611Z     * 26322544b8 (base)  14 reports, total time  4545.22s ± 1821.39s
2022-11-23T03:00:41.2310057Z     * 7f4b4d2827         14 reports, total time  4184.38s ± 2086.66s
2022-11-23T03:00:41.2310535Z     * b50699f247         14 reports, total time  4471.77s ± 1861.64s
2022-11-23T03:00:41.2311551Z     * 8bf8e4d71e         14 reports, total time  4570.74s ± 1842.55s
2022-11-23T03:00:41.2312513Z     * ce856cee7e         14 reports, total time  4577.38s ± 1838.96s
2022-11-23T03:00:41.2313414Z     * 391b593ca2         14 reports, total time  4457.06s ± 1822.27s
2022-11-23T03:00:41.2313879Z     * 5bba783d21         14 reports, total time  4489.09s ± 1854.90s
2022-11-23T03:00:41.2314279Z     * ea920a1115         14 reports, total time  4455.13s ± 1838.12s
2022-11-23T03:00:41.2314704Z     * 74e62a1fef         14 reports, total time  4552.71s ± 1811.91s
2022-11-23T03:00:41.2315128Z     * 00b7d8ef23         14 reports, total time  4180.04s ± 2065.84s
2022-11-23T03:00:41.2315382Z     |
2022-11-23T03:00:41.2320336Z     :
2022-11-23T03:00:41.2320532Z 
2022-11-23T03:00:41.2320720Z Removed  (across  996 suites)     0 tests, totaling     0.00s
2022-11-23T03:00:41.2321095Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-11-23T03:00:41.2321447Z Added    (across   75 suites)   876 tests, totaling +3943.05s
2022-11-23T03:00:41.2891804Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2022-11-23T03:00:41.2892145Z with:
2022-11-23T03:00:41.2892360Z env:
2022-11-23T03:00:41.2892599Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:41.2892852Z   GPU_FLAG: --gpus all
2022-11-23T03:00:41.2893214Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:41.2893559Z ##[endgroup]
2022-11-23T03:00:41.2911948Z ##[group]Run set -eou pipefail
2022-11-23T03:00:41.2912258Z set -eou pipefail
2022-11-23T03:00:41.2912506Z 
2022-11-23T03:00:41.2912824Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-11-23T03:00:41.2913144Z for _ in $(seq 1440); do
2022-11-23T03:00:41.2913438Z     # Break if no ssh session exists anymore
2022-11-23T03:00:41.2913881Z     if [ "$(who)" = "" ]; then
2022-11-23T03:00:41.2914117Z       break
2022-11-23T03:00:41.2914387Z     fi
2022-11-23T03:00:41.2914621Z     echo "."
2022-11-23T03:00:41.2914844Z     sleep 5
2022-11-23T03:00:41.2915078Z done
2022-11-23T03:00:41.2928539Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:00:41.2928821Z env:
2022-11-23T03:00:41.2929064Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:41.2929330Z   GPU_FLAG: --gpus all
2022-11-23T03:00:41.2929673Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:41.2930021Z ##[endgroup]
2022-11-23T03:00:41.2960010Z Holding runner for 2 hours until all ssh sessions have logged out
2022-11-23T03:00:41.3024415Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T03:00:41.3024846Z # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T03:00:41.3025196Z # shellcheck disable=SC2046
2022-11-23T03:00:41.3025505Z docker stop $(docker ps -q) || true
2022-11-23T03:00:41.3025807Z # Prune all of the docker images
2022-11-23T03:00:41.3026106Z docker system prune -af
2022-11-23T03:00:41.3038448Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:00:41.3038742Z env:
2022-11-23T03:00:41.3038982Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:00:41.3039229Z   GPU_FLAG: --gpus all
2022-11-23T03:00:41.3039587Z   DOCKER_CONTAINER_ID: 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:41.3039930Z ##[endgroup]
2022-11-23T03:00:42.1937026Z 846842cde4cc
2022-11-23T03:00:43.2534403Z Deleted Containers:
2022-11-23T03:00:48.3885141Z 846842cde4cc4d04758896fb11de9f1f3068d3fd4852d4d9fe87a943f9edd77e
2022-11-23T03:00:48.3885560Z 
2022-11-23T03:00:48.3885768Z Deleted Images:
2022-11-23T03:00:48.3886680Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T03:00:48.3887674Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:3a5626edfb2c43fb24303351be75287af92426b6bb7c6df2defc98f980346c6a
2022-11-23T03:00:48.3888287Z deleted: sha256:e2c63e8434298b5b8922fe396fb22d541e83da3321f8559334df676354c6a90a
2022-11-23T03:00:48.3888722Z deleted: sha256:e97e2654456ae35786d9ff4a73ece4d85ce36ae9bd4e402e5f8c4c41a4b8cb5d
2022-11-23T03:00:48.3889169Z deleted: sha256:0191afefc9967131b7cd6196bee5a1d3a4eba8c24d3e11ff67013ecd0d244f4d
2022-11-23T03:00:48.3889575Z deleted: sha256:cd6998962c740e934e511d315fd0139a2737289173123cd7675b630fe71d0a6f
2022-11-23T03:00:48.3889993Z deleted: sha256:684c9dbbc4faf4388438a99012caaa6e9e9c3ac93f3842ff7b2f4c81c6c66866
2022-11-23T03:00:48.3890436Z deleted: sha256:be75865fc66b386df8a53dd220b7f4fa8464d0c86f06b6fa84e7d5b8fa2b5333
2022-11-23T03:00:48.3890853Z deleted: sha256:9e5281171ccc5aa329fd085f38d4831c13f47e27ea26a9243daf336fc701114a
2022-11-23T03:00:48.3891277Z deleted: sha256:0ba6072392ef0b01b99d45293e62f415e397460b4bf5a00257afb7aa9cfccb14
2022-11-23T03:00:48.3891886Z deleted: sha256:5f0fab79723550908a4149737ce5268ceacba20bc9c1aea35acdb6ff93ba4aa7
2022-11-23T03:00:48.3892312Z deleted: sha256:0c8088138816657b983280a5e4385f5c159c90b6be095bc4972290be20d46c16
2022-11-23T03:00:48.3892700Z deleted: sha256:a9cdd96267ff8adf28efa06db7d37977216a7580ca475239528fff85024f9bcb
2022-11-23T03:00:48.3893132Z deleted: sha256:9abd11e0b20ee19055f20e11ac5a4cc029eee3433686ce8ab9ffb6636269391a
2022-11-23T03:00:48.3893564Z deleted: sha256:cb16cc59b9c802a04fe3283c4a00840d0a3c24128b3620964a7aa927a757d672
2022-11-23T03:00:48.3893965Z deleted: sha256:ed27e40372acea88785f25bcd63f03a56960f00e444e3d5b22e52915e885242b
2022-11-23T03:00:48.3894412Z deleted: sha256:395dfa2cf9efd2fde511c14dbaf706e2efb3ab003af0cd725614b86f10643247
2022-11-23T03:00:48.3894845Z deleted: sha256:ca415181cb076083a9af8e85b901ee24154183e2d4c3960e21aab48260376214
2022-11-23T03:00:48.3895368Z deleted: sha256:b13fc2861b47406c24208813cb5398b911d9bae952f11ed9a411f42e221f8dfc
2022-11-23T03:00:48.3895783Z deleted: sha256:9cbf0b121bab50c1cad2d31b40f6c7c52003ba77877a2ef6d9bc87a2c0b073d2
2022-11-23T03:00:48.3896237Z deleted: sha256:60e157b04ecdbe2bce04795e0fade9ec9aae999065bd410785dcbaedd9778a19
2022-11-23T03:00:48.3896657Z deleted: sha256:5eb96691864f520823a417cd2f3278b4c2ac579490941d6c623865e478828c8b
2022-11-23T03:00:48.3897038Z deleted: sha256:e93d6940ac64ac73f178cc63066fb2c3ab041023d66146b32019cb7860511be5
2022-11-23T03:00:48.3897446Z deleted: sha256:e302e1f04c7e3031f83227f08d6987b02f39a75ac0e741754afad2dc1e265f8a
2022-11-23T03:00:48.3897859Z deleted: sha256:d82cdf793dbcd047c1843326443a1249721e7308a7c6fb3e23fe7331652e7047
2022-11-23T03:00:48.3898547Z deleted: sha256:3edb430c2f9009d4993daf017be01fe272bd3452db11c16e51f7755ac845d410
2022-11-23T03:00:48.3898958Z deleted: sha256:16e8f362c1784e16c1db6b1d0aa4449097e6d646f4c8682a122dea7c4da38aaf
2022-11-23T03:00:48.3899379Z deleted: sha256:7f58576cf19df9f3be9082f2c0ec2fc7010409b97ecb99bae66a10805d752f48
2022-11-23T03:00:48.3899813Z deleted: sha256:88688611a15825ecae20cd8c4032711d2351d2f954a9ebcd4c671b2bdb017df8
2022-11-23T03:00:48.3900219Z deleted: sha256:a46e0b74ccdcd4e2eb07727be3bc1a2c4236b1f88c65e64a50234e8a35932a80
2022-11-23T03:00:48.3900660Z deleted: sha256:b633962159aa14dfe94a149d00f90eecaba6dab960d4011bdf3667a5ee9586db
2022-11-23T03:00:48.3901106Z deleted: sha256:a05c7409499ce8c5d7ffc085772c3910c812ec835dc9145bbbb07b8b3c075235
2022-11-23T03:00:48.3901531Z deleted: sha256:0d63a7de5066f69cd9fd1af8fc47405e880de8f88f5cb16278a1f1ac94d0cd41
2022-11-23T03:00:48.3901927Z deleted: sha256:7d74b4ce1a60334100fccc0917345873714e640160622691b579d64c0ae4640f
2022-11-23T03:00:48.3902334Z deleted: sha256:33aae29ffc4791507bef289cfb1f178909f3fc97a40c618723eeec1f8f5bd80c
2022-11-23T03:00:48.3902774Z deleted: sha256:6ea72b84f0436ed1d288baf124dd38e43bbb89e746ccfd3a4ec420ddced8bbc2
2022-11-23T03:00:48.3903221Z deleted: sha256:04e33e1cfdd5a1b2409b80f5881e6cf7b1810fe975aad4ce7c97b0ff6c0e7b4e
2022-11-23T03:00:48.3903651Z deleted: sha256:df1ef30e86bf04681ecd0728263efe1e98b2eea0a228cef29bd0febfc8bdac2f
2022-11-23T03:00:48.3904078Z deleted: sha256:36a44974e500014175f5e49f50c8afa1ac9c5e8092a8ea99c3c97b7ce9c517d8
2022-11-23T03:00:48.3904495Z deleted: sha256:a31f0224d50d031823b07dbb97f256f6960c87ea3c52ebeceef98febab200451
2022-11-23T03:00:48.3904908Z deleted: sha256:c4beec84548d277aff0487a9a5a8c2b3d577421e3275f36106b778c6edbb9d53
2022-11-23T03:00:48.3905327Z deleted: sha256:bcc7df3b45729f5d1802045954e76e3407d9e07ba6f516de0895d775d00ad7f8
2022-11-23T03:00:48.3905754Z deleted: sha256:84de992a179a16ba619507ec45b04b4c0da3d3fa31cedc8f6beb5aaadd7a232a
2022-11-23T03:00:48.3906179Z deleted: sha256:5011206a0b2edc2a6c68ba41313e7f283ee7c925ab6a731f8818d01352f68596
2022-11-23T03:00:48.3906582Z deleted: sha256:46a56b12ac94daa35c90ac97d26adfde704693e34613d69fb97687aa53ae33f5
2022-11-23T03:00:48.3907013Z deleted: sha256:ed2b7a9e28b3474bc9b7e68f8158ecda88b3fa3d3ab1587898fa976922af0deb
2022-11-23T03:00:48.3907445Z deleted: sha256:4a6976746db7764bb48f2a06af1fb5f88e3646edc1c9bc0d18686d5a6350cac0
2022-11-23T03:00:48.3907946Z deleted: sha256:5e175425e3e9ec93e8c6c1b7560b49ef5e95af68ec55757902072a8dca020323
2022-11-23T03:00:48.3908373Z deleted: sha256:fb740502513c6cf883c844f03760de367c4c70d09a69b9476bcf737b4578563a
2022-11-23T03:00:48.3908779Z deleted: sha256:2c105119fc030d11b3d570ec9a83948a1fb17f138df2a3245f9566b89de51495
2022-11-23T03:00:48.3909210Z deleted: sha256:8caad6b6cba0d0ced7e21fe4b2027b8647d66b7f78c34367dd8571a0520ba2c0
2022-11-23T03:00:48.3909643Z deleted: sha256:1051db32aefad193995ca536ed99e29eed4fd0340ddda721ec11e9c4eb9e93af
2022-11-23T03:00:48.3910076Z deleted: sha256:c6b2a4553f41b3b4a3dc6a26be0020c98980bb4e7186d194901769dce6716c27
2022-11-23T03:00:48.3910512Z deleted: sha256:8faec3528fe75bb31f14d0caf8707a2fe4b70f60d7e631c2b3dbb36cd6d83dd9
2022-11-23T03:00:48.3910929Z deleted: sha256:7574bc80094251ac667e6bed9dd5a808ecf6f61f23c8d4c56a69c644d06f4e32
2022-11-23T03:00:48.3911346Z deleted: sha256:69f57fbceb1b420d7e4697e0f6514887b0805ee0059bea7d51e0a832962e74bf
2022-11-23T03:00:48.3911667Z 
2022-11-23T03:00:48.4009750Z Total reclaimed space: 21.93GB
2022-11-23T03:00:48.4068881Z Post job cleanup.
2022-11-23T03:00:48.4138891Z Post job cleanup.
2022-11-23T03:00:48.5489790Z [command]/usr/bin/git version
2022-11-23T03:00:48.5544141Z git version 2.37.1
2022-11-23T03:00:48.5606973Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/759daf87-76e6-460d-a546-4a4cf188c8b7' before making global git config changes
2022-11-23T03:00:48.5607521Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T03:00:48.5613752Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T03:00:48.5730869Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T03:00:48.5768453Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T03:00:48.6089731Z Entering 'android/libs/fbjni'
2022-11-23T03:00:48.6132031Z Entering 'third_party/FP16'
2022-11-23T03:00:48.6174659Z Entering 'third_party/FXdiv'
2022-11-23T03:00:48.6216943Z Entering 'third_party/NNPACK'
2022-11-23T03:00:48.6258248Z Entering 'third_party/QNNPACK'
2022-11-23T03:00:48.6300001Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T03:00:48.6342527Z Entering 'third_party/XNNPACK'
2022-11-23T03:00:48.6395378Z Entering 'third_party/benchmark'
2022-11-23T03:00:48.6437832Z Entering 'third_party/cpuinfo'
2022-11-23T03:00:48.6479744Z Entering 'third_party/cub'
2022-11-23T03:00:48.6520847Z Entering 'third_party/cudnn_frontend'
2022-11-23T03:00:48.6567969Z Entering 'third_party/cutlass'
2022-11-23T03:00:48.6616964Z Entering 'third_party/eigen'
2022-11-23T03:00:48.6661110Z Entering 'third_party/fbgemm'
2022-11-23T03:00:48.6704146Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T03:00:48.6747251Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T03:00:48.6789096Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T03:00:48.6830117Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T03:00:48.6872493Z Entering 'third_party/flatbuffers'
2022-11-23T03:00:48.6917061Z Entering 'third_party/fmt'
2022-11-23T03:00:48.6958682Z Entering 'third_party/foxi'
2022-11-23T03:00:48.7000064Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T03:00:48.7041940Z Entering 'third_party/gloo'
2022-11-23T03:00:48.7083898Z Entering 'third_party/googletest'
2022-11-23T03:00:48.7125274Z Entering 'third_party/ideep'
2022-11-23T03:00:48.7166552Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T03:00:48.7210100Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T03:00:48.7259465Z Entering 'third_party/ios-cmake'
2022-11-23T03:00:48.7301911Z Entering 'third_party/ittapi'
2022-11-23T03:00:48.7343997Z Entering 'third_party/kineto'
2022-11-23T03:00:48.7387152Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T03:00:48.7429022Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T03:00:48.7471894Z Entering 'third_party/nccl/nccl'
2022-11-23T03:00:48.7513630Z Entering 'third_party/neon2sse'
2022-11-23T03:00:48.7555723Z Entering 'third_party/nlohmann'
2022-11-23T03:00:48.7599715Z Entering 'third_party/onnx'
2022-11-23T03:00:48.7655055Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T03:00:48.7697400Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T03:00:48.7742880Z Entering 'third_party/onnx-tensorrt'
2022-11-23T03:00:48.7784259Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T03:00:48.7829944Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T03:00:48.7872237Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T03:00:48.7913799Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T03:00:48.7960602Z Entering 'third_party/pocketfft'
2022-11-23T03:00:48.8002234Z Entering 'third_party/protobuf'
2022-11-23T03:00:48.8048570Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T03:00:48.8089973Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T03:00:48.8133033Z Entering 'third_party/psimd'
2022-11-23T03:00:48.8175140Z Entering 'third_party/pthreadpool'
2022-11-23T03:00:48.8216456Z Entering 'third_party/pybind11'
2022-11-23T03:00:48.8258165Z Entering 'third_party/python-enum'
2022-11-23T03:00:48.8299002Z Entering 'third_party/python-peachpy'
2022-11-23T03:00:48.8340336Z Entering 'third_party/python-six'
2022-11-23T03:00:48.8381900Z Entering 'third_party/sleef'
2022-11-23T03:00:48.8423822Z Entering 'third_party/tbb'
2022-11-23T03:00:48.8468807Z Entering 'third_party/tensorpipe'
2022-11-23T03:00:48.8511558Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T03:00:48.8561699Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T03:00:48.8604822Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T03:00:48.8647078Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T03:00:48.8688344Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T03:00:48.8733730Z Entering 'third_party/zstd'
2022-11-23T03:00:48.8794462Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T03:00:48.8823665Z http.https://github.com/.extraheader
2022-11-23T03:00:48.8833077Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-11-23T03:00:48.8868682Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T03:00:48.9185159Z Entering 'android/libs/fbjni'
2022-11-23T03:00:48.9209204Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9243020Z Entering 'third_party/FP16'
2022-11-23T03:00:48.9268499Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9300271Z Entering 'third_party/FXdiv'
2022-11-23T03:00:48.9324715Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9357752Z Entering 'third_party/NNPACK'
2022-11-23T03:00:48.9384143Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9416962Z Entering 'third_party/QNNPACK'
2022-11-23T03:00:48.9442165Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9477679Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T03:00:48.9505623Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9547461Z Entering 'third_party/XNNPACK'
2022-11-23T03:00:48.9571701Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9615941Z Entering 'third_party/benchmark'
2022-11-23T03:00:48.9641334Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9674199Z Entering 'third_party/cpuinfo'
2022-11-23T03:00:48.9698333Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9730854Z Entering 'third_party/cub'
2022-11-23T03:00:48.9756042Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9788321Z Entering 'third_party/cudnn_frontend'
2022-11-23T03:00:48.9812454Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9852304Z Entering 'third_party/cutlass'
2022-11-23T03:00:48.9877352Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9918212Z Entering 'third_party/eigen'
2022-11-23T03:00:48.9943594Z http.https://github.com/.extraheader
2022-11-23T03:00:48.9977071Z Entering 'third_party/fbgemm'
2022-11-23T03:00:49.0001704Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0034277Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T03:00:49.0058356Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0090757Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T03:00:49.0116598Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0150943Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T03:00:49.0175228Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0206881Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T03:00:49.0231598Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0266349Z Entering 'third_party/flatbuffers'
2022-11-23T03:00:49.0291236Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0325265Z Entering 'third_party/fmt'
2022-11-23T03:00:49.0350831Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0383378Z Entering 'third_party/foxi'
2022-11-23T03:00:49.0407519Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0438789Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T03:00:49.0464041Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0495628Z Entering 'third_party/gloo'
2022-11-23T03:00:49.0520053Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0567736Z Entering 'third_party/googletest'
2022-11-23T03:00:49.0592470Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0625775Z Entering 'third_party/ideep'
2022-11-23T03:00:49.0649519Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0680734Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T03:00:49.0705118Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0740756Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T03:00:49.0764965Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0804749Z Entering 'third_party/ios-cmake'
2022-11-23T03:00:49.0829418Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0861963Z Entering 'third_party/ittapi'
2022-11-23T03:00:49.0886987Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0920393Z Entering 'third_party/kineto'
2022-11-23T03:00:49.0944935Z http.https://github.com/.extraheader
2022-11-23T03:00:49.0976861Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T03:00:49.1002365Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1034264Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T03:00:49.1057863Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1091572Z Entering 'third_party/nccl/nccl'
2022-11-23T03:00:49.1116127Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1148945Z Entering 'third_party/neon2sse'
2022-11-23T03:00:49.1173919Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1207216Z Entering 'third_party/nlohmann'
2022-11-23T03:00:49.1232860Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1266704Z Entering 'third_party/onnx'
2022-11-23T03:00:49.1302912Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1350176Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T03:00:49.1374616Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1408072Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T03:00:49.1432651Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1467218Z Entering 'third_party/onnx-tensorrt'
2022-11-23T03:00:49.1491624Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1522896Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T03:00:49.1548102Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1585656Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T03:00:49.1609641Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1642197Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T03:00:49.1667120Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1699875Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T03:00:49.1723971Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1760871Z Entering 'third_party/pocketfft'
2022-11-23T03:00:49.1785400Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1816635Z Entering 'third_party/protobuf'
2022-11-23T03:00:49.1841595Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1876711Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T03:00:49.1901362Z http.https://github.com/.extraheader
2022-11-23T03:00:49.1952969Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T03:00:49.1977349Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2010987Z Entering 'third_party/psimd'
2022-11-23T03:00:49.2036276Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2069004Z Entering 'third_party/pthreadpool'
2022-11-23T03:00:49.2093289Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2125529Z Entering 'third_party/pybind11'
2022-11-23T03:00:49.2150966Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2184024Z Entering 'third_party/python-enum'
2022-11-23T03:00:49.2208073Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2240057Z Entering 'third_party/python-peachpy'
2022-11-23T03:00:49.2265274Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2296621Z Entering 'third_party/python-six'
2022-11-23T03:00:49.2320972Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2352623Z Entering 'third_party/sleef'
2022-11-23T03:00:49.2376211Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2407959Z Entering 'third_party/tbb'
2022-11-23T03:00:49.2432824Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2466835Z Entering 'third_party/tensorpipe'
2022-11-23T03:00:49.2490984Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2524409Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T03:00:49.2548268Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2580449Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T03:00:49.2605166Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2638049Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T03:00:49.2662181Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2695767Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T03:00:49.2719602Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2752047Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T03:00:49.2776283Z http.https://github.com/.extraheader
2022-11-23T03:00:49.2812370Z Entering 'third_party/zstd'
2022-11-23T03:00:49.2837612Z http.https://github.com/.extraheader
2022-11-23T03:00:49.3132219Z Cleaning up orphan processes