2022-11-23T01:10:21.9479978Z Requested labels: linux.8xlarge.nvidia.gpu
2022-11-23T01:10:21.9480048Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-11-23T01:10:21.9480071Z Waiting for a runner to pick up this job...
2022-11-23T01:10:22.2406329Z Job is about to start running on the runner: i-0b134addf6413be7c (organization)
2022-11-23T01:10:26.5287611Z Current runner version: '2.299.1'
2022-11-23T01:10:26.5295615Z Runner name: 'i-0b134addf6413be7c'
2022-11-23T01:10:26.5296407Z Runner group name: 'Default'
2022-11-23T01:10:26.5297199Z Machine name: 'ip-10-0-4-63'
2022-11-23T01:10:26.5299877Z ##[group]GITHUB_TOKEN Permissions
2022-11-23T01:10:26.5300840Z Actions: write
2022-11-23T01:10:26.5301324Z Checks: write
2022-11-23T01:10:26.5301695Z Contents: write
2022-11-23T01:10:26.5302130Z Deployments: write
2022-11-23T01:10:26.5302626Z Discussions: write
2022-11-23T01:10:26.5303014Z Issues: write
2022-11-23T01:10:26.5303435Z Metadata: read
2022-11-23T01:10:26.5303865Z Packages: write
2022-11-23T01:10:26.5304235Z Pages: write
2022-11-23T01:10:26.5304731Z PullRequests: write
2022-11-23T01:10:26.5305325Z RepositoryProjects: write
2022-11-23T01:10:26.5305756Z SecurityEvents: write
2022-11-23T01:10:26.5306199Z Statuses: write
2022-11-23T01:10:26.5306637Z ##[endgroup]
2022-11-23T01:10:26.5310816Z Secret source: Actions
2022-11-23T01:10:26.5311798Z Prepare workflow directory
2022-11-23T01:10:26.8685539Z Prepare all required actions
2022-11-23T01:10:26.8925420Z Getting action download info
2022-11-23T01:10:27.0798320Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd)
2022-11-23T01:10:27.4976359Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f)
2022-11-23T01:10:30.7571944Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2022-11-23T01:10:31.1337943Z Getting action download info
2022-11-23T01:10:31.2833418Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-11-23T01:10:32.3224169Z Getting action download info
2022-11-23T01:10:32.4642256Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-11-23T01:10:32.6640082Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-11-23T01:10:32.6642702Z ##[group] Inputs
2022-11-23T01:10:32.6643097Z   build-environment: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:10:32.6644705Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "slow", shard: 1, num_shards: 2, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "slow", shard: 2, num_shards: 2, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "nogpu_AVX512", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "nogpu_NO_AVX2", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
]}

2022-11-23T01:10:32.6646581Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:10:32.6647074Z   sync-tag: 
2022-11-23T01:10:32.6648160Z   timeout-minutes: 240
2022-11-23T01:10:32.6648455Z ##[endgroup]
2022-11-23T01:10:32.6649280Z Complete job name: linux-bionic-cuda11.7-py3.10-gcc7 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu)
2022-11-23T01:10:32.7780850Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2022-11-23T01:10:32.7781230Z with:
2022-11-23T01:10:32.7781807Z   github-secret: ***
2022-11-23T01:10:32.7782108Z   activate-with-label: false
2022-11-23T01:10:32.7782364Z   label: with-ssh
2022-11-23T01:10:32.7782805Z   remove-existing-keys: true
2022-11-23T01:10:32.7783076Z env:
2022-11-23T01:10:32.7783305Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:32.7783571Z ##[endgroup]
2022-11-23T01:10:32.8856385Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-11-23T01:10:32.9087531Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-11-23T01:10:32.9087890Z with:
2022-11-23T01:10:32.9088136Z   submodules: recursive
2022-11-23T01:10:32.9088395Z   fetch-depth: 0
2022-11-23T01:10:32.9088610Z env:
2022-11-23T01:10:32.9088871Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:32.9089178Z ##[endgroup]
2022-11-23T01:10:32.9378967Z ##[group]Run retry () {
2022-11-23T01:10:32.9379296Z retry () {
2022-11-23T01:10:32.9379620Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-11-23T01:10:32.9379904Z }
2022-11-23T01:10:32.9380165Z echo "${GITHUB_WORKSPACE}"
2022-11-23T01:10:32.9380482Z if [ -z "${NO_SUDO}" ]; then
2022-11-23T01:10:32.9380779Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:10:32.9381064Z else
2022-11-23T01:10:32.9381343Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:10:32.9381593Z fi
2022-11-23T01:10:32.9381890Z mkdir "${GITHUB_WORKSPACE}"
2022-11-23T01:10:32.9400915Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:10:32.9401232Z env:
2022-11-23T01:10:32.9401485Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:32.9401741Z   NO_SUDO: 
2022-11-23T01:10:32.9401962Z ##[endgroup]
2022-11-23T01:10:32.9527664Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:35.9169644Z ##[group]Run malfet/checkout@silent-checkout
2022-11-23T01:10:35.9169989Z with:
2022-11-23T01:10:35.9170262Z   ref: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:10:35.9170528Z   fetch-depth: 0
2022-11-23T01:10:35.9170786Z   submodules: recursive
2022-11-23T01:10:35.9171051Z   quiet-checkout: true
2022-11-23T01:10:35.9171324Z   repository: pytorch/pytorch
2022-11-23T01:10:35.9172098Z   token: ***
2022-11-23T01:10:35.9172347Z   ssh-strict: true
2022-11-23T01:10:35.9172599Z   persist-credentials: true
2022-11-23T01:10:35.9172865Z   clean: true
2022-11-23T01:10:35.9173103Z   lfs: false
2022-11-23T01:10:35.9173341Z   set-safe-directory: true
2022-11-23T01:10:35.9173590Z env:
2022-11-23T01:10:35.9173830Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:35.9174072Z ##[endgroup]
2022-11-23T01:10:36.0679920Z Syncing repository: pytorch/pytorch
2022-11-23T01:10:36.0681745Z ##[group]Getting Git version info
2022-11-23T01:10:36.0682296Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:10:36.0682901Z [command]/usr/bin/git version
2022-11-23T01:10:36.0683158Z git version 2.37.1
2022-11-23T01:10:36.0693482Z ##[endgroup]
2022-11-23T01:10:36.0713081Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/f00beadf-dd26-4fb1-99fd-f137092d63aa' before making global git config changes
2022-11-23T01:10:36.0714392Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T01:10:36.0720351Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:36.0766174Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:10:36.0773023Z ##[group]Initializing the repository
2022-11-23T01:10:36.0776713Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:36.0809075Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-11-23T01:10:36.0809534Z hint: is subject to change. To configure the initial branch name to use in all
2022-11-23T01:10:36.0809949Z hint: of your new repositories, which will suppress this warning, call:
2022-11-23T01:10:36.0810268Z hint: 
2022-11-23T01:10:36.0810648Z hint: 	git config --global init.defaultBranch <name>
2022-11-23T01:10:36.0811128Z hint: 
2022-11-23T01:10:36.0811724Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-11-23T01:10:36.0812236Z hint: 'development'. The just-created branch can be renamed via this command:
2022-11-23T01:10:36.0812549Z hint: 
2022-11-23T01:10:36.0812937Z hint: 	git branch -m <name>
2022-11-23T01:10:36.0814282Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-11-23T01:10:36.0824744Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-11-23T01:10:36.0860886Z ##[endgroup]
2022-11-23T01:10:36.0861386Z ##[group]Disabling automatic garbage collection
2022-11-23T01:10:36.0865342Z [command]/usr/bin/git config --local gc.auto 0
2022-11-23T01:10:36.0897106Z ##[endgroup]
2022-11-23T01:10:36.0898071Z ##[group]Setting up auth
2022-11-23T01:10:36.0907238Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T01:10:36.0943602Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T01:10:36.1245618Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T01:10:36.1276643Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T01:10:36.1578210Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:10:36.1625495Z ##[endgroup]
2022-11-23T01:10:36.1625981Z ##[group]Fetching the repository
2022-11-23T01:10:36.1634221Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-11-23T01:11:31.2665619Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object}
2022-11-23T01:11:31.2696638Z 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:11:31.2704431Z ##[endgroup]
2022-11-23T01:11:31.2705097Z ##[group]Determining the checkout info
2022-11-23T01:11:31.2705548Z ##[endgroup]
2022-11-23T01:11:31.2705998Z ##[group]Checking out the ref
2022-11-23T01:11:31.2709394Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:11:33.0213698Z ##[endgroup]
2022-11-23T01:11:33.0214272Z ##[group]Setting up auth for fetching submodules
2022-11-23T01:11:33.0220297Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:11:33.0275751Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-11-23T01:11:33.0309026Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-11-23T01:11:33.0342236Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-11-23T01:11:33.0374753Z ##[endgroup]
2022-11-23T01:11:33.0375201Z ##[group]Fetching submodules
2022-11-23T01:11:33.0380123Z [command]/usr/bin/git submodule sync --recursive
2022-11-23T01:11:33.0706911Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-11-23T01:11:33.1019872Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-11-23T01:11:33.1022481Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-11-23T01:11:33.1025461Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-11-23T01:11:33.1028647Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-11-23T01:11:33.1031987Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-11-23T01:11:33.1035602Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-11-23T01:11:33.1039207Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-11-23T01:11:33.1042786Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-11-23T01:11:33.1046635Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-11-23T01:11:33.1050528Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-11-23T01:11:33.1056534Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-11-23T01:11:33.1061536Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-11-23T01:11:33.1066675Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-11-23T01:11:33.1072130Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-11-23T01:11:33.1077155Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-11-23T01:11:33.1082493Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-11-23T01:11:33.1087969Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-11-23T01:11:33.1094936Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:11:33.1100587Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-11-23T01:11:33.1106829Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-11-23T01:11:33.1112986Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-11-23T01:11:33.1119311Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-11-23T01:11:33.1125444Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-11-23T01:11:33.1131805Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-11-23T01:11:33.1138931Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-11-23T01:11:33.1145777Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-11-23T01:11:33.1152412Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-11-23T01:11:33.1159260Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-11-23T01:11:33.1166694Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-11-23T01:11:33.1174432Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-11-23T01:11:33.1181889Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-11-23T01:11:33.1189373Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-11-23T01:11:33.1196927Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-11-23T01:11:33.1204809Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-11-23T01:11:33.1212865Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-11-23T01:11:33.1221168Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-11-23T01:11:33.1229183Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-11-23T01:11:33.1237353Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-11-23T01:11:33.1245961Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-11-23T01:11:33.1255515Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-11-23T01:11:33.1263966Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-11-23T01:11:33.1292544Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-11-23T01:11:33.4191097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-11-23T01:11:33.6313148Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-11-23T01:11:33.8442275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-11-23T01:11:34.4103010Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-11-23T01:11:34.6958859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-11-23T01:11:36.6008147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-11-23T01:11:42.3461645Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-11-23T01:11:42.7801751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-11-23T01:11:43.3246827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-11-23T01:11:44.8546061Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-11-23T01:11:46.1650110Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-11-23T01:11:47.8113068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-11-23T01:11:54.8120307Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-11-23T01:11:55.5552123Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-11-23T01:11:57.0479741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-11-23T01:11:58.1659886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-11-23T01:11:58.3636049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-11-23T01:11:58.8464720Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-11-23T01:11:59.1536252Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-11-23T01:12:00.1416636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-11-23T01:12:00.5796284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-11-23T01:12:00.7976343Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-11-23T01:12:01.0981934Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-11-23T01:12:02.7869624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-11-23T01:12:03.2699090Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-11-23T01:12:03.6776016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-11-23T01:12:09.9122721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-11-23T01:12:11.6680053Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-11-23T01:12:12.1002616Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-11-23T01:12:12.3439139Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-11-23T01:12:18.3753630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-11-23T01:12:18.5794599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-11-23T01:12:18.8043163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-11-23T01:12:19.6766177Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-11-23T01:12:19.9033798Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-11-23T01:12:20.3184314Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-11-23T01:12:20.6573069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-11-23T01:12:21.2250831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-11-23T01:12:23.7291211Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-11-23T01:12:24.2653467Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-11-23T01:12:26.5290239Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-11-23T01:12:26.5412911Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-11-23T01:12:26.5507519Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-11-23T01:12:26.5786057Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-11-23T01:12:26.6052660Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-11-23T01:12:26.6488626Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-11-23T01:12:27.4006973Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-11-23T01:12:27.4257749Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:12:27.5480599Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-11-23T01:12:27.5881349Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-11-23T01:12:27.9467297Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-11-23T01:12:28.4657820Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-11-23T01:12:28.7701262Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-11-23T01:12:28.8275054Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a'
2022-11-23T01:12:28.8292470Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:12:28.8296284Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:12:28.8299611Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:12:28.8303210Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:12:28.8330241Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-11-23T01:12:29.7943774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-11-23T01:12:30.3578232Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-11-23T01:12:31.3592189Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-11-23T01:12:31.6949209Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-11-23T01:12:31.8203490Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-11-23T01:12:31.8924889Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-11-23T01:12:31.9040409Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-11-23T01:12:32.0222040Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-11-23T01:12:32.0660140Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2022-11-23T01:12:32.0763509Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-11-23T01:12:32.1256172Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-11-23T01:12:32.1555049Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2022-11-23T01:12:32.2123846Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-11-23T01:12:32.2259145Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f'
2022-11-23T01:12:32.2275844Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-11-23T01:12:32.2302570Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-11-23T01:12:40.8164567Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d'
2022-11-23T01:12:40.8184768Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:12:40.8212757Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-11-23T01:12:49.6438328Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289'
2022-11-23T01:12:49.6557620Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-11-23T01:12:49.6728712Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-11-23T01:12:49.7856654Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2022-11-23T01:12:49.7874766Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:12:49.7877946Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:12:49.7905494Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-11-23T01:12:50.9348015Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-11-23T01:12:52.0048302Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-11-23T01:12:52.0706378Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-11-23T01:12:52.0953586Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-11-23T01:12:52.1115436Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-11-23T01:12:52.2449661Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-11-23T01:12:52.5634897Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-11-23T01:12:52.5667641Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-11-23T01:12:52.5670718Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-11-23T01:12:52.5699074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-11-23T01:12:52.9802649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-11-23T01:12:53.8562173Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:12:53.8942172Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-11-23T01:12:53.9120404Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-11-23T01:12:53.9137879Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:12:53.9163939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-11-23T01:12:55.7714501Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-11-23T01:12:55.7737254Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:12:55.7740405Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:12:55.7767842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-11-23T01:12:56.1889158Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-11-23T01:12:57.0709287Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-11-23T01:12:57.1493512Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-11-23T01:12:57.1510085Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:12:57.1537297Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-11-23T01:12:57.7394065Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:12:57.7498148Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-11-23T01:12:58.0696269Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-11-23T01:12:58.0719023Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:12:58.0722452Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-11-23T01:12:58.0750581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-11-23T01:12:58.5073110Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-11-23T01:12:59.5300272Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-11-23T01:12:59.6135154Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-11-23T01:12:59.6232816Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-11-23T01:12:59.6359182Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-11-23T01:12:59.6764142Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2022-11-23T01:12:59.6864911Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-11-23T01:12:59.7213569Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-11-23T01:12:59.7320859Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-11-23T01:12:59.7863447Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-11-23T01:12:59.9239473Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-11-23T01:12:59.9555646Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-11-23T01:12:59.9573091Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:12:59.9576436Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:12:59.9579571Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:12:59.9582934Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:12:59.9609120Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-11-23T01:13:00.9652878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-11-23T01:13:01.2824197Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-11-23T01:13:02.6470522Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-11-23T01:13:03.9517757Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-11-23T01:13:03.9684560Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-11-23T01:13:04.0465661Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-11-23T01:13:04.0799131Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-11-23T01:13:04.0815282Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:04.0842087Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-11-23T01:13:04.3237146Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:13:04.4844854Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-11-23T01:13:04.4886164Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-11-23T01:13:04.5214885Z Entering 'android/libs/fbjni'
2022-11-23T01:13:04.5259754Z Entering 'third_party/FP16'
2022-11-23T01:13:04.5301769Z Entering 'third_party/FXdiv'
2022-11-23T01:13:04.5344643Z Entering 'third_party/NNPACK'
2022-11-23T01:13:04.5386543Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:04.5430200Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:04.5473905Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:04.5528519Z Entering 'third_party/benchmark'
2022-11-23T01:13:04.5571384Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:04.5615419Z Entering 'third_party/cub'
2022-11-23T01:13:04.5658826Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:04.5709357Z Entering 'third_party/cutlass'
2022-11-23T01:13:04.5758211Z Entering 'third_party/eigen'
2022-11-23T01:13:04.5803903Z Entering 'third_party/fbgemm'
2022-11-23T01:13:04.5846670Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:04.5888271Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:04.5932199Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:04.5975727Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:04.6019359Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:04.6064113Z Entering 'third_party/fmt'
2022-11-23T01:13:04.6106175Z Entering 'third_party/foxi'
2022-11-23T01:13:04.6148419Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:04.6190878Z Entering 'third_party/gloo'
2022-11-23T01:13:04.6235623Z Entering 'third_party/googletest'
2022-11-23T01:13:04.6279504Z Entering 'third_party/ideep'
2022-11-23T01:13:04.6321739Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:04.6365969Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:04.6415871Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:04.6458611Z Entering 'third_party/ittapi'
2022-11-23T01:13:04.6501519Z Entering 'third_party/kineto'
2022-11-23T01:13:04.6543289Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:04.6585090Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:04.6628971Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:04.6671397Z Entering 'third_party/neon2sse'
2022-11-23T01:13:04.6714251Z Entering 'third_party/nlohmann'
2022-11-23T01:13:04.6757290Z Entering 'third_party/onnx'
2022-11-23T01:13:04.6812337Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:04.6855070Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:04.6899681Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:04.6941387Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:04.6988812Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:04.7030594Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:04.7072108Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:04.7119681Z Entering 'third_party/pocketfft'
2022-11-23T01:13:04.7162535Z Entering 'third_party/protobuf'
2022-11-23T01:13:04.7208864Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:04.7251209Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:04.7296011Z Entering 'third_party/psimd'
2022-11-23T01:13:04.7338345Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:04.7380777Z Entering 'third_party/pybind11'
2022-11-23T01:13:04.7424356Z Entering 'third_party/python-enum'
2022-11-23T01:13:04.7465767Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:04.7507531Z Entering 'third_party/python-six'
2022-11-23T01:13:04.7549472Z Entering 'third_party/sleef'
2022-11-23T01:13:04.7592083Z Entering 'third_party/tbb'
2022-11-23T01:13:04.7636509Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:04.7680121Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:04.7722441Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:04.7763906Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:04.7807370Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:04.7849221Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:04.7894491Z Entering 'third_party/zstd'
2022-11-23T01:13:04.7950466Z ##[endgroup]
2022-11-23T01:13:04.7951001Z ##[group]Persisting credentials for submodules
2022-11-23T01:13:04.7955624Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-11-23T01:13:04.8275123Z Entering 'android/libs/fbjni'
2022-11-23T01:13:04.8317962Z Entering 'third_party/FP16'
2022-11-23T01:13:04.8360079Z Entering 'third_party/FXdiv'
2022-11-23T01:13:04.8401734Z Entering 'third_party/NNPACK'
2022-11-23T01:13:04.8443521Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:04.8485900Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:04.8527750Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:04.8581721Z Entering 'third_party/benchmark'
2022-11-23T01:13:04.8623169Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:04.8665654Z Entering 'third_party/cub'
2022-11-23T01:13:04.8707066Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:04.8754512Z Entering 'third_party/cutlass'
2022-11-23T01:13:04.8803628Z Entering 'third_party/eigen'
2022-11-23T01:13:04.8846985Z Entering 'third_party/fbgemm'
2022-11-23T01:13:04.8889062Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:04.8930171Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:04.8972273Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:04.9013343Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:04.9056792Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:04.9100456Z Entering 'third_party/fmt'
2022-11-23T01:13:04.9141929Z Entering 'third_party/foxi'
2022-11-23T01:13:04.9183305Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:04.9224830Z Entering 'third_party/gloo'
2022-11-23T01:13:04.9265903Z Entering 'third_party/googletest'
2022-11-23T01:13:04.9307455Z Entering 'third_party/ideep'
2022-11-23T01:13:04.9348490Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:04.9391555Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:04.9439614Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:04.9481564Z Entering 'third_party/ittapi'
2022-11-23T01:13:04.9523348Z Entering 'third_party/kineto'
2022-11-23T01:13:04.9565270Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:04.9606422Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:04.9649367Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:04.9691407Z Entering 'third_party/neon2sse'
2022-11-23T01:13:04.9734210Z Entering 'third_party/nlohmann'
2022-11-23T01:13:04.9776585Z Entering 'third_party/onnx'
2022-11-23T01:13:04.9830699Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:04.9873222Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:04.9919083Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:04.9959738Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:05.0005384Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.0047697Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.0089027Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.0137772Z Entering 'third_party/pocketfft'
2022-11-23T01:13:05.0179602Z Entering 'third_party/protobuf'
2022-11-23T01:13:05.0225003Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:05.0267224Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:05.0309380Z Entering 'third_party/psimd'
2022-11-23T01:13:05.0351088Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:05.0393671Z Entering 'third_party/pybind11'
2022-11-23T01:13:05.0435459Z Entering 'third_party/python-enum'
2022-11-23T01:13:05.0477652Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:05.0518981Z Entering 'third_party/python-six'
2022-11-23T01:13:05.0560788Z Entering 'third_party/sleef'
2022-11-23T01:13:05.0602489Z Entering 'third_party/tbb'
2022-11-23T01:13:05.0645997Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:05.0687482Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:05.0728806Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:05.0769423Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:05.0810763Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:05.0851076Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.0897126Z Entering 'third_party/zstd'
2022-11-23T01:13:05.0951712Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-11-23T01:13:05.1266847Z Entering 'android/libs/fbjni'
2022-11-23T01:13:05.1305216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-11-23T01:13:05.1323486Z Entering 'third_party/FP16'
2022-11-23T01:13:05.1362814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-11-23T01:13:05.1380760Z Entering 'third_party/FXdiv'
2022-11-23T01:13:05.1420066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-11-23T01:13:05.1437069Z Entering 'third_party/NNPACK'
2022-11-23T01:13:05.1475940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-11-23T01:13:05.1494374Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:05.1533263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-11-23T01:13:05.1551095Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:05.1590463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-11-23T01:13:05.1608083Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:05.1646995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-11-23T01:13:05.1675610Z Entering 'third_party/benchmark'
2022-11-23T01:13:05.1715132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:05.1733551Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:05.1771799Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:13:05.1789831Z Entering 'third_party/cub'
2022-11-23T01:13:05.1828675Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-11-23T01:13:05.1846567Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:05.1886355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-11-23T01:13:05.1909774Z Entering 'third_party/cutlass'
2022-11-23T01:13:05.1950354Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-11-23T01:13:05.1974455Z Entering 'third_party/eigen'
2022-11-23T01:13:05.2012184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-11-23T01:13:05.2031621Z Entering 'third_party/fbgemm'
2022-11-23T01:13:05.2071435Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-11-23T01:13:05.2088302Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:05.2126807Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-11-23T01:13:05.2145170Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:05.2183512Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:13:05.2201553Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:05.2240779Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.2259179Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:05.2297655Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-11-23T01:13:05.2315977Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:05.2354702Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-11-23T01:13:05.2374597Z Entering 'third_party/fmt'
2022-11-23T01:13:05.2412583Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-11-23T01:13:05.2430450Z Entering 'third_party/foxi'
2022-11-23T01:13:05.2469235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-11-23T01:13:05.2486365Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:05.2525284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-11-23T01:13:05.2543314Z Entering 'third_party/gloo'
2022-11-23T01:13:05.2582937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-11-23T01:13:05.2600285Z Entering 'third_party/googletest'
2022-11-23T01:13:05.2640031Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.2658262Z Entering 'third_party/ideep'
2022-11-23T01:13:05.2697185Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-11-23T01:13:05.2713616Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:05.2752760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-11-23T01:13:05.2771930Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:05.2810861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-11-23T01:13:05.2836011Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:05.2876265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-11-23T01:13:05.2893939Z Entering 'third_party/ittapi'
2022-11-23T01:13:05.2931874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-11-23T01:13:05.2950805Z Entering 'third_party/kineto'
2022-11-23T01:13:05.2990347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-11-23T01:13:05.3007656Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:05.3047192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-11-23T01:13:05.3066191Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:05.3104636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.3122966Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:05.3162223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-11-23T01:13:05.3181634Z Entering 'third_party/neon2sse'
2022-11-23T01:13:05.3220968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-11-23T01:13:05.3237937Z Entering 'third_party/nlohmann'
2022-11-23T01:13:05.3278658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-11-23T01:13:05.3298203Z Entering 'third_party/onnx'
2022-11-23T01:13:05.3337569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:13:05.3366825Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.3406443Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:05.3424344Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.3463715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:05.3483845Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:05.3522954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-11-23T01:13:05.3540131Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:05.3578733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:13:05.3601097Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.3640431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:05.3658639Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.3698102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:05.3714994Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.3755260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:13:05.3778203Z Entering 'third_party/pocketfft'
2022-11-23T01:13:05.3817202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-11-23T01:13:05.3834607Z Entering 'third_party/protobuf'
2022-11-23T01:13:05.3874041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-11-23T01:13:05.3895694Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:05.3934878Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:05.3952335Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:05.3992146Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.4011264Z Entering 'third_party/psimd'
2022-11-23T01:13:05.4050795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-11-23T01:13:05.4068720Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:05.4108688Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-11-23T01:13:05.4125961Z Entering 'third_party/pybind11'
2022-11-23T01:13:05.4165907Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:05.4184810Z Entering 'third_party/python-enum'
2022-11-23T01:13:05.4223636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-11-23T01:13:05.4241110Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:05.4280767Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-11-23T01:13:05.4298320Z Entering 'third_party/python-six'
2022-11-23T01:13:05.4337996Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-11-23T01:13:05.4354744Z Entering 'third_party/sleef'
2022-11-23T01:13:05.4394356Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-11-23T01:13:05.4412008Z Entering 'third_party/tbb'
2022-11-23T01:13:05.4450216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-11-23T01:13:05.4470294Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:05.4509986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-11-23T01:13:05.4527902Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:05.4566961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:05.4584947Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:05.4623681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-11-23T01:13:05.4640462Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:05.4679418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-11-23T01:13:05.4697230Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:05.4736391Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:05.4752629Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.4791946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:13:05.4812492Z Entering 'third_party/zstd'
2022-11-23T01:13:05.4850461Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-11-23T01:13:05.5657628Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-11-23T01:13:05.5976432Z Entering 'android/libs/fbjni'
2022-11-23T01:13:05.6018241Z Entering 'third_party/FP16'
2022-11-23T01:13:05.6061678Z Entering 'third_party/FXdiv'
2022-11-23T01:13:05.6103806Z Entering 'third_party/NNPACK'
2022-11-23T01:13:05.6147755Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:05.6192971Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:05.6236694Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:05.6290927Z Entering 'third_party/benchmark'
2022-11-23T01:13:05.6335102Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:05.6378283Z Entering 'third_party/cub'
2022-11-23T01:13:05.6421108Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:05.6470500Z Entering 'third_party/cutlass'
2022-11-23T01:13:05.6520076Z Entering 'third_party/eigen'
2022-11-23T01:13:05.6566223Z Entering 'third_party/fbgemm'
2022-11-23T01:13:05.6608967Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:05.6651206Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:05.6694656Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:05.6737541Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:05.6780538Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:05.6824923Z Entering 'third_party/fmt'
2022-11-23T01:13:05.6867554Z Entering 'third_party/foxi'
2022-11-23T01:13:05.6909622Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:05.6953228Z Entering 'third_party/gloo'
2022-11-23T01:13:05.6995869Z Entering 'third_party/googletest'
2022-11-23T01:13:05.7039475Z Entering 'third_party/ideep'
2022-11-23T01:13:05.7081687Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:05.7126685Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:05.7176489Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:05.7218912Z Entering 'third_party/ittapi'
2022-11-23T01:13:05.7262477Z Entering 'third_party/kineto'
2022-11-23T01:13:05.7304808Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:05.7347895Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:05.7391815Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:05.7435130Z Entering 'third_party/neon2sse'
2022-11-23T01:13:05.7478040Z Entering 'third_party/nlohmann'
2022-11-23T01:13:05.7522025Z Entering 'third_party/onnx'
2022-11-23T01:13:05.7578388Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.7621232Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.7665680Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:05.7707747Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:05.7754602Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:05.7797372Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:05.7840477Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.7888502Z Entering 'third_party/pocketfft'
2022-11-23T01:13:05.7932572Z Entering 'third_party/protobuf'
2022-11-23T01:13:05.7980471Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:05.8024034Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:05.8068913Z Entering 'third_party/psimd'
2022-11-23T01:13:05.8113386Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:05.8156932Z Entering 'third_party/pybind11'
2022-11-23T01:13:05.8200201Z Entering 'third_party/python-enum'
2022-11-23T01:13:05.8243729Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:05.8286793Z Entering 'third_party/python-six'
2022-11-23T01:13:05.8330109Z Entering 'third_party/sleef'
2022-11-23T01:13:05.8374407Z Entering 'third_party/tbb'
2022-11-23T01:13:05.8419782Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:05.8462893Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:05.8504980Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:05.8547416Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:05.8589749Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:05.8631683Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:05.8677156Z Entering 'third_party/zstd'
2022-11-23T01:13:05.8732346Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-11-23T01:13:05.9047101Z Entering 'android/libs/fbjni'
2022-11-23T01:13:05.9089964Z Entering 'third_party/FP16'
2022-11-23T01:13:05.9133855Z Entering 'third_party/FXdiv'
2022-11-23T01:13:05.9176729Z Entering 'third_party/NNPACK'
2022-11-23T01:13:05.9219738Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:05.9262860Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:05.9306006Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:05.9361003Z Entering 'third_party/benchmark'
2022-11-23T01:13:05.9404945Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:05.9448727Z Entering 'third_party/cub'
2022-11-23T01:13:05.9492906Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:05.9542520Z Entering 'third_party/cutlass'
2022-11-23T01:13:05.9592842Z Entering 'third_party/eigen'
2022-11-23T01:13:05.9639124Z Entering 'third_party/fbgemm'
2022-11-23T01:13:05.9682623Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:05.9725326Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:05.9768632Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:05.9811137Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:05.9854659Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:05.9900248Z Entering 'third_party/fmt'
2022-11-23T01:13:05.9943508Z Entering 'third_party/foxi'
2022-11-23T01:13:05.9986878Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:06.0029249Z Entering 'third_party/gloo'
2022-11-23T01:13:06.0072554Z Entering 'third_party/googletest'
2022-11-23T01:13:06.0115391Z Entering 'third_party/ideep'
2022-11-23T01:13:06.0157992Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:06.0202338Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:06.0252712Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:06.0295842Z Entering 'third_party/ittapi'
2022-11-23T01:13:06.0337702Z Entering 'third_party/kineto'
2022-11-23T01:13:06.0380723Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:06.0422676Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:06.0466529Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:06.0509219Z Entering 'third_party/neon2sse'
2022-11-23T01:13:06.0551953Z Entering 'third_party/nlohmann'
2022-11-23T01:13:06.0596789Z Entering 'third_party/onnx'
2022-11-23T01:13:06.0653068Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:06.0696948Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:06.0741846Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:06.0784082Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:06.0830669Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:06.0873994Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:06.0917142Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:06.0964529Z Entering 'third_party/pocketfft'
2022-11-23T01:13:06.1007677Z Entering 'third_party/protobuf'
2022-11-23T01:13:06.1055636Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:06.1097525Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:06.1142058Z Entering 'third_party/psimd'
2022-11-23T01:13:06.1184043Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:06.1226388Z Entering 'third_party/pybind11'
2022-11-23T01:13:06.1268379Z Entering 'third_party/python-enum'
2022-11-23T01:13:06.1311849Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:06.1354209Z Entering 'third_party/python-six'
2022-11-23T01:13:06.1397194Z Entering 'third_party/sleef'
2022-11-23T01:13:06.1440485Z Entering 'third_party/tbb'
2022-11-23T01:13:06.1485870Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:06.1530400Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:06.1573804Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:06.1616070Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:06.1659532Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:06.1701314Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:06.1746128Z Entering 'third_party/zstd'
2022-11-23T01:13:06.1798735Z ##[endgroup]
2022-11-23T01:13:06.1853843Z [command]/usr/bin/git log -1 --format='%H'
2022-11-23T01:13:06.1883054Z '1cfd3858ac54fe3883534309081631a0a892ba3f'
2022-11-23T01:13:06.2031351Z Prepare all required actions
2022-11-23T01:13:06.2062698Z ##[group]Run ./.github/actions/setup-linux
2022-11-23T01:13:06.2062986Z env:
2022-11-23T01:13:06.2063234Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:06.2063480Z ##[endgroup]
2022-11-23T01:13:06.2123951Z ##[group]Run set -euo pipefail
2022-11-23T01:13:06.2124244Z set -euo pipefail
2022-11-23T01:13:06.2124517Z function get_ec2_metadata() {
2022-11-23T01:13:06.2124857Z   # Pulled from instance metadata endpoint for EC2
2022-11-23T01:13:06.2125333Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-11-23T01:13:06.2125721Z   category=$1
2022-11-23T01:13:06.2126025Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-11-23T01:13:06.2126321Z }
2022-11-23T01:13:06.2126587Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-11-23T01:13:06.2126935Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-11-23T01:13:06.2127296Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-11-23T01:13:06.2127612Z echo "system info $(uname -a)"
2022-11-23T01:13:06.2141722Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:06.2142006Z env:
2022-11-23T01:13:06.2142226Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:06.2142486Z ##[endgroup]
2022-11-23T01:13:06.2243728Z ami-id: ami-096198a0bccc6bad4
2022-11-23T01:13:06.2306541Z instance-id: i-0b134addf6413be7c
2022-11-23T01:13:06.2368510Z instance-type: g3.8xlarge
2022-11-23T01:13:06.2379208Z system info Linux ip-10-0-4-63.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-11-23T01:13:06.2397697Z ##[group]Run if systemctl is-active --quiet docker; then
2022-11-23T01:13:06.2398085Z if systemctl is-active --quiet docker; then
2022-11-23T01:13:06.2398423Z     echo "Docker daemon is running...";
2022-11-23T01:13:06.2398707Z else
2022-11-23T01:13:06.2399015Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-11-23T01:13:06.2399330Z fi
2022-11-23T01:13:06.2411958Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:06.2412246Z env:
2022-11-23T01:13:06.2412489Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:06.2412751Z ##[endgroup]
2022-11-23T01:13:06.2464556Z Docker daemon is running...
2022-11-23T01:13:06.2483232Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:13:06.2483696Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:13:06.2484084Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:06.2484586Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-11-23T01:13:06.2485058Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-11-23T01:13:06.2497446Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:06.2497750Z env:
2022-11-23T01:13:06.2497997Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:06.2498269Z   AWS_RETRY_MODE: standard
2022-11-23T01:13:06.2498512Z   AWS_MAX_ATTEMPTS: 5
2022-11-23T01:13:06.2498787Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T01:13:06.2499046Z ##[endgroup]
2022-11-23T01:13:07.1946567Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-11-23T01:13:07.1947053Z Configure a credential helper to remove this warning. See
2022-11-23T01:13:07.1947584Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-11-23T01:13:07.1947860Z 
2022-11-23T01:13:07.1948976Z Login Succeeded
2022-11-23T01:13:07.1984760Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:07.1985188Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:07.1985674Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:07.1999039Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:07.1999328Z env:
2022-11-23T01:13:07.1999572Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:07.1999841Z ##[endgroup]
2022-11-23T01:13:07.2090256Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2022-11-23T01:13:07.2090628Z with:
2022-11-23T01:13:07.2091115Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:13:07.2092119Z env:
2022-11-23T01:13:07.2092368Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:07.2092628Z ##[endgroup]
2022-11-23T01:13:07.2109282Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:07.2109670Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:07.2110038Z # ignore output since only exit code is used for conditional
2022-11-23T01:13:07.2110427Z # only pull docker image if it's not available locally
2022-11-23T01:13:07.2110834Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-11-23T01:13:07.2111233Z   retry docker pull "${DOCKER_IMAGE}"
2022-11-23T01:13:07.2111510Z fi
2022-11-23T01:13:07.2124011Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:07.2124313Z env:
2022-11-23T01:13:07.2124556Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:07.2125053Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:13:07.2125534Z ##[endgroup]
2022-11-23T01:13:07.4415618Z 072aae4a77ed7d3a69ad5683420509c41301b940: Pulling from pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7
2022-11-23T01:13:07.4416759Z a404e5416296: Pulling fs layer
2022-11-23T01:13:07.4417291Z 42d021f37342: Pulling fs layer
2022-11-23T01:13:07.4417731Z 9dab8401a678: Pulling fs layer
2022-11-23T01:13:07.4418926Z 2fc72180b8eb: Pulling fs layer
2022-11-23T01:13:07.4419217Z 16e6a4d496ed: Pulling fs layer
2022-11-23T01:13:07.4419479Z 3a4c0b092a9e: Pulling fs layer
2022-11-23T01:13:07.4419773Z d9e19734e968: Pulling fs layer
2022-11-23T01:13:07.4420053Z 8859cc6b3ab7: Pulling fs layer
2022-11-23T01:13:07.4420310Z 1a4b44db2103: Pulling fs layer
2022-11-23T01:13:07.4420584Z fac00e927cfe: Pulling fs layer
2022-11-23T01:13:07.4425071Z 2133f4081ddb: Pulling fs layer
2022-11-23T01:13:07.4425873Z 3ceac802dd07: Pulling fs layer
2022-11-23T01:13:07.4426518Z 69c929400d35: Pulling fs layer
2022-11-23T01:13:07.4427034Z bc2be817cb7e: Pulling fs layer
2022-11-23T01:13:07.4427573Z d04926f1c68d: Pulling fs layer
2022-11-23T01:13:07.4428072Z 91f116f19c0d: Pulling fs layer
2022-11-23T01:13:07.4428606Z a7cf5b3894f8: Pulling fs layer
2022-11-23T01:13:07.4428895Z 722cddd823f9: Pulling fs layer
2022-11-23T01:13:07.4429148Z 2e882087d824: Pulling fs layer
2022-11-23T01:13:07.4429552Z ba6235196410: Pulling fs layer
2022-11-23T01:13:07.4430111Z 2fc72180b8eb: Waiting
2022-11-23T01:13:07.4430577Z 313c5ee380ab: Pulling fs layer
2022-11-23T01:13:07.4431072Z 16e6a4d496ed: Waiting
2022-11-23T01:13:07.4431548Z 6ff0fc00b0a9: Pulling fs layer
2022-11-23T01:13:07.4432001Z 06f043acdbfb: Pulling fs layer
2022-11-23T01:13:07.4432443Z a8c562f6a1cf: Pulling fs layer
2022-11-23T01:13:07.4432890Z 3a4c0b092a9e: Waiting
2022-11-23T01:13:07.4433362Z 4468cd4f574c: Pulling fs layer
2022-11-23T01:13:07.4433841Z 46e5d3cec398: Pulling fs layer
2022-11-23T01:13:07.4434323Z bc2be817cb7e: Waiting
2022-11-23T01:13:07.4434747Z 165006759af3: Pulling fs layer
2022-11-23T01:13:07.4435193Z d9e19734e968: Waiting
2022-11-23T01:13:07.4435626Z 2133f4081ddb: Waiting
2022-11-23T01:13:07.4436268Z 3ceac802dd07: Waiting
2022-11-23T01:13:07.4436645Z 2c660101e145: Pulling fs layer
2022-11-23T01:13:07.4437081Z 8ce731125a4e: Pulling fs layer
2022-11-23T01:13:07.4437485Z 8859cc6b3ab7: Waiting
2022-11-23T01:13:07.4437865Z 102ddcd90753: Pulling fs layer
2022-11-23T01:13:07.4438290Z 0cff5716a932: Pulling fs layer
2022-11-23T01:13:07.4438702Z 1a4b44db2103: Waiting
2022-11-23T01:13:07.4439044Z 91f116f19c0d: Waiting
2022-11-23T01:13:07.4439402Z fac00e927cfe: Waiting
2022-11-23T01:13:07.4439928Z 867a6ccb577f: Pulling fs layer
2022-11-23T01:13:07.4440345Z 863c35620b44: Pulling fs layer
2022-11-23T01:13:07.4440739Z 722cddd823f9: Waiting
2022-11-23T01:13:07.4441097Z 4468cd4f574c: Waiting
2022-11-23T01:13:07.4441465Z f828f00c6d66: Pulling fs layer
2022-11-23T01:13:07.4441846Z 46e5d3cec398: Waiting
2022-11-23T01:13:07.4442278Z f0fe61569b0b: Pulling fs layer
2022-11-23T01:13:07.4442650Z a9f3d4742233: Pulling fs layer
2022-11-23T01:13:07.4443114Z 000b6751ea6f: Pulling fs layer
2022-11-23T01:13:07.4443461Z a8c562f6a1cf: Waiting
2022-11-23T01:13:07.4443688Z 165006759af3: Waiting
2022-11-23T01:13:07.4443924Z 2e882087d824: Waiting
2022-11-23T01:13:07.4444347Z 6ff0fc00b0a9: Waiting
2022-11-23T01:13:07.4444671Z ba6235196410: Waiting
2022-11-23T01:13:07.4445023Z 023a41fa48e6: Pulling fs layer
2022-11-23T01:13:07.4445457Z 081025f05026: Pulling fs layer
2022-11-23T01:13:07.4445719Z a7cf5b3894f8: Waiting
2022-11-23T01:13:07.4445961Z 5970defc1d8b: Pulling fs layer
2022-11-23T01:13:07.4446258Z 7e2d6313145f: Pulling fs layer
2022-11-23T01:13:07.4446759Z 4b4d66451d67: Pulling fs layer
2022-11-23T01:13:07.4447151Z 75f1ead35ace: Pulling fs layer
2022-11-23T01:13:07.4447424Z 793c37004dab: Pulling fs layer
2022-11-23T01:13:07.4447679Z 8ce731125a4e: Waiting
2022-11-23T01:13:07.4447901Z 863c35620b44: Waiting
2022-11-23T01:13:07.4448153Z 4f1313d71da0: Pulling fs layer
2022-11-23T01:13:07.4448423Z 6386b2adbe28: Pulling fs layer
2022-11-23T01:13:07.4448678Z f21c39caec7a: Pulling fs layer
2022-11-23T01:13:07.4448957Z 867a6ccb577f: Waiting
2022-11-23T01:13:07.4449224Z 2baaae32e3d5: Pulling fs layer
2022-11-23T01:13:07.4449463Z 023a41fa48e6: Waiting
2022-11-23T01:13:07.4449716Z 128ebe6811c5: Pulling fs layer
2022-11-23T01:13:07.4449966Z 2c660101e145: Waiting
2022-11-23T01:13:07.4450187Z 4b4d66451d67: Waiting
2022-11-23T01:13:07.4450425Z f828f00c6d66: Waiting
2022-11-23T01:13:07.4450666Z 5970defc1d8b: Waiting
2022-11-23T01:13:07.4450887Z 793c37004dab: Waiting
2022-11-23T01:13:07.4451129Z f21c39caec7a: Waiting
2022-11-23T01:13:07.4451372Z 000b6751ea6f: Waiting
2022-11-23T01:13:07.4452102Z 772fa4efddc3: Pulling fs layer
2022-11-23T01:13:07.4452377Z 2baaae32e3d5: Waiting
2022-11-23T01:13:07.4452619Z 6386b2adbe28: Waiting
2022-11-23T01:13:07.4452855Z c36ac12376c5: Pulling fs layer
2022-11-23T01:13:07.4453127Z 85cc9b957510: Pulling fs layer
2022-11-23T01:13:07.4453395Z 9ab0826d88b5: Pulling fs layer
2022-11-23T01:13:07.4453632Z 85cc9b957510: Waiting
2022-11-23T01:13:07.4453874Z 772fa4efddc3: Waiting
2022-11-23T01:13:07.4454113Z 9ab0826d88b5: Waiting
2022-11-23T01:13:07.4454356Z 102ddcd90753: Waiting
2022-11-23T01:13:07.4454573Z c36ac12376c5: Waiting
2022-11-23T01:13:07.4454809Z 128ebe6811c5: Waiting
2022-11-23T01:13:07.4455040Z 081025f05026: Waiting
2022-11-23T01:13:07.4455255Z f0fe61569b0b: Waiting
2022-11-23T01:13:07.5801375Z 42d021f37342: Verifying Checksum
2022-11-23T01:13:07.5801695Z 42d021f37342: Download complete
2022-11-23T01:13:07.6649516Z 2fc72180b8eb: Verifying Checksum
2022-11-23T01:13:07.6649832Z 2fc72180b8eb: Download complete
2022-11-23T01:13:07.7628901Z 16e6a4d496ed: Verifying Checksum
2022-11-23T01:13:07.7629240Z 16e6a4d496ed: Download complete
2022-11-23T01:13:07.8161301Z a404e5416296: Download complete
2022-11-23T01:13:07.8917251Z d9e19734e968: Download complete
2022-11-23T01:13:07.9825828Z 9dab8401a678: Verifying Checksum
2022-11-23T01:13:07.9826146Z 9dab8401a678: Download complete
2022-11-23T01:13:08.0595822Z 1a4b44db2103: Verifying Checksum
2022-11-23T01:13:08.0596191Z 1a4b44db2103: Download complete
2022-11-23T01:13:08.1427984Z fac00e927cfe: Verifying Checksum
2022-11-23T01:13:08.1428799Z fac00e927cfe: Download complete
2022-11-23T01:13:08.8905215Z a404e5416296: Pull complete
2022-11-23T01:13:09.1727778Z 42d021f37342: Pull complete
2022-11-23T01:13:10.1107653Z 9dab8401a678: Pull complete
2022-11-23T01:13:10.2013563Z 2133f4081ddb: Verifying Checksum
2022-11-23T01:13:10.2013959Z 2133f4081ddb: Download complete
2022-11-23T01:13:10.2214234Z 2fc72180b8eb: Pull complete
2022-11-23T01:13:10.2703783Z 3ceac802dd07: Verifying Checksum
2022-11-23T01:13:10.2704363Z 3ceac802dd07: Download complete
2022-11-23T01:13:10.3470767Z 69c929400d35: Download complete
2022-11-23T01:13:10.3734751Z 16e6a4d496ed: Pull complete
2022-11-23T01:13:10.4272165Z bc2be817cb7e: Verifying Checksum
2022-11-23T01:13:10.4273240Z bc2be817cb7e: Download complete
2022-11-23T01:13:11.1509157Z d04926f1c68d: Verifying Checksum
2022-11-23T01:13:11.1509798Z d04926f1c68d: Download complete
2022-11-23T01:13:11.2259736Z 91f116f19c0d: Verifying Checksum
2022-11-23T01:13:11.2260082Z 91f116f19c0d: Download complete
2022-11-23T01:13:11.3264272Z a7cf5b3894f8: Verifying Checksum
2022-11-23T01:13:11.3264627Z a7cf5b3894f8: Download complete
2022-11-23T01:13:18.6807436Z 3a4c0b092a9e: Verifying Checksum
2022-11-23T01:13:18.6807780Z 3a4c0b092a9e: Download complete
2022-11-23T01:13:18.7643617Z 2e882087d824: Download complete
2022-11-23T01:13:18.8460102Z ba6235196410: Verifying Checksum
2022-11-23T01:13:18.8460716Z ba6235196410: Download complete
2022-11-23T01:13:18.9378448Z 313c5ee380ab: Download complete
2022-11-23T01:13:19.0119219Z 6ff0fc00b0a9: Download complete
2022-11-23T01:13:19.0831033Z 06f043acdbfb: Verifying Checksum
2022-11-23T01:13:19.0831654Z 06f043acdbfb: Download complete
2022-11-23T01:13:19.1677014Z a8c562f6a1cf: Verifying Checksum
2022-11-23T01:13:19.1677633Z a8c562f6a1cf: Download complete
2022-11-23T01:13:20.1178269Z 4468cd4f574c: Verifying Checksum
2022-11-23T01:13:20.1178725Z 4468cd4f574c: Download complete
2022-11-23T01:13:20.2016020Z 46e5d3cec398: Verifying Checksum
2022-11-23T01:13:20.2016376Z 46e5d3cec398: Download complete
2022-11-23T01:13:20.2886235Z 165006759af3: Verifying Checksum
2022-11-23T01:13:20.2886922Z 165006759af3: Download complete
2022-11-23T01:13:20.3886971Z 2c660101e145: Verifying Checksum
2022-11-23T01:13:20.3887333Z 2c660101e145: Download complete
2022-11-23T01:13:20.4573203Z 8ce731125a4e: Verifying Checksum
2022-11-23T01:13:20.4573547Z 8ce731125a4e: Download complete
2022-11-23T01:13:20.5305840Z 102ddcd90753: Verifying Checksum
2022-11-23T01:13:20.5306969Z 102ddcd90753: Download complete
2022-11-23T01:13:22.0729973Z 8859cc6b3ab7: Verifying Checksum
2022-11-23T01:13:22.0730479Z 8859cc6b3ab7: Download complete
2022-11-23T01:13:22.1698914Z 867a6ccb577f: Verifying Checksum
2022-11-23T01:13:22.1699360Z 867a6ccb577f: Download complete
2022-11-23T01:13:22.2472717Z 863c35620b44: Verifying Checksum
2022-11-23T01:13:22.2473073Z 863c35620b44: Download complete
2022-11-23T01:13:22.5120109Z 0cff5716a932: Verifying Checksum
2022-11-23T01:13:22.5120472Z 0cff5716a932: Download complete
2022-11-23T01:13:22.5824651Z f0fe61569b0b: Verifying Checksum
2022-11-23T01:13:22.5824965Z f0fe61569b0b: Download complete
2022-11-23T01:13:22.6335333Z f828f00c6d66: Verifying Checksum
2022-11-23T01:13:22.6335826Z f828f00c6d66: Download complete
2022-11-23T01:13:22.6577405Z a9f3d4742233: Verifying Checksum
2022-11-23T01:13:22.6577711Z a9f3d4742233: Download complete
2022-11-23T01:13:22.7386913Z 023a41fa48e6: Download complete
2022-11-23T01:13:22.9039147Z 000b6751ea6f: Verifying Checksum
2022-11-23T01:13:22.9039550Z 000b6751ea6f: Download complete
2022-11-23T01:13:22.9828582Z 5970defc1d8b: Verifying Checksum
2022-11-23T01:13:22.9828925Z 5970defc1d8b: Download complete
2022-11-23T01:13:23.0605368Z 7e2d6313145f: Verifying Checksum
2022-11-23T01:13:23.0605702Z 7e2d6313145f: Download complete
2022-11-23T01:13:23.1868555Z 081025f05026: Verifying Checksum
2022-11-23T01:13:23.1869150Z 081025f05026: Download complete
2022-11-23T01:13:23.2585763Z 75f1ead35ace: Verifying Checksum
2022-11-23T01:13:23.2586418Z 75f1ead35ace: Download complete
2022-11-23T01:13:23.3536525Z 793c37004dab: Download complete
2022-11-23T01:13:23.4454322Z 4f1313d71da0: Download complete
2022-11-23T01:13:23.5361090Z 6386b2adbe28: Download complete
2022-11-23T01:13:23.7134687Z f21c39caec7a: Verifying Checksum
2022-11-23T01:13:23.7135228Z f21c39caec7a: Download complete
2022-11-23T01:13:23.7976715Z 2baaae32e3d5: Verifying Checksum
2022-11-23T01:13:23.7977233Z 2baaae32e3d5: Download complete
2022-11-23T01:13:24.4067562Z 128ebe6811c5: Verifying Checksum
2022-11-23T01:13:24.4068138Z 128ebe6811c5: Download complete
2022-11-23T01:13:24.4862237Z 772fa4efddc3: Download complete
2022-11-23T01:13:27.7689565Z 4b4d66451d67: Verifying Checksum
2022-11-23T01:13:27.7690085Z 4b4d66451d67: Download complete
2022-11-23T01:13:27.8539031Z 85cc9b957510: Download complete
2022-11-23T01:13:27.9335250Z 9ab0826d88b5: Verifying Checksum
2022-11-23T01:13:27.9335747Z 9ab0826d88b5: Download complete
2022-11-23T01:13:31.5998774Z 722cddd823f9: Verifying Checksum
2022-11-23T01:13:31.5999439Z 722cddd823f9: Download complete
2022-11-23T01:13:32.9447113Z 3a4c0b092a9e: Pull complete
2022-11-23T01:13:33.0695128Z d9e19734e968: Pull complete
2022-11-23T01:13:47.4558828Z c36ac12376c5: Verifying Checksum
2022-11-23T01:13:47.4559196Z c36ac12376c5: Download complete
2022-11-23T01:13:54.8483378Z 8859cc6b3ab7: Pull complete
2022-11-23T01:13:56.6958177Z 1a4b44db2103: Pull complete
2022-11-23T01:13:58.5777029Z fac00e927cfe: Pull complete
2022-11-23T01:14:06.6297642Z 2133f4081ddb: Pull complete
2022-11-23T01:14:08.5400441Z 3ceac802dd07: Pull complete
2022-11-23T01:14:10.5440478Z 69c929400d35: Pull complete
2022-11-23T01:14:13.8550119Z bc2be817cb7e: Pull complete
2022-11-23T01:14:17.5538011Z d04926f1c68d: Pull complete
2022-11-23T01:14:19.6319991Z 91f116f19c0d: Pull complete
2022-11-23T01:14:21.9933350Z a7cf5b3894f8: Pull complete
2022-11-23T01:14:56.5761403Z 722cddd823f9: Pull complete
2022-11-23T01:14:56.7022607Z 2e882087d824: Pull complete
2022-11-23T01:14:56.8137184Z ba6235196410: Pull complete
2022-11-23T01:14:56.9112928Z 313c5ee380ab: Pull complete
2022-11-23T01:14:57.0059844Z 6ff0fc00b0a9: Pull complete
2022-11-23T01:14:57.1419471Z 06f043acdbfb: Pull complete
2022-11-23T01:14:57.2224262Z a8c562f6a1cf: Pull complete
2022-11-23T01:14:59.5642789Z 4468cd4f574c: Pull complete
2022-11-23T01:14:59.6734977Z 46e5d3cec398: Pull complete
2022-11-23T01:14:59.7861612Z 165006759af3: Pull complete
2022-11-23T01:14:59.9724002Z 2c660101e145: Pull complete
2022-11-23T01:15:00.1188830Z 8ce731125a4e: Pull complete
2022-11-23T01:15:00.2459315Z 102ddcd90753: Pull complete
2022-11-23T01:15:08.3918059Z 0cff5716a932: Pull complete
2022-11-23T01:15:10.2625696Z 867a6ccb577f: Pull complete
2022-11-23T01:15:12.1724168Z 863c35620b44: Pull complete
2022-11-23T01:15:15.1732011Z f828f00c6d66: Pull complete
2022-11-23T01:15:17.0470601Z f0fe61569b0b: Pull complete
2022-11-23T01:15:18.9213567Z a9f3d4742233: Pull complete
2022-11-23T01:15:21.2806448Z 000b6751ea6f: Pull complete
2022-11-23T01:15:24.7544796Z 023a41fa48e6: Pull complete
2022-11-23T01:15:29.7985665Z 081025f05026: Pull complete
2022-11-23T01:15:32.0095127Z 5970defc1d8b: Pull complete
2022-11-23T01:15:35.3954137Z 7e2d6313145f: Pull complete
2022-11-23T01:15:43.7538043Z 4b4d66451d67: Pull complete
2022-11-23T01:15:46.3112478Z 75f1ead35ace: Pull complete
2022-11-23T01:15:48.2179786Z 793c37004dab: Pull complete
2022-11-23T01:15:50.1112252Z 4f1313d71da0: Pull complete
2022-11-23T01:15:52.2946316Z 6386b2adbe28: Pull complete
2022-11-23T01:15:53.7115742Z f21c39caec7a: Pull complete
2022-11-23T01:15:53.8184095Z 2baaae32e3d5: Pull complete
2022-11-23T01:15:55.7333490Z 128ebe6811c5: Pull complete
2022-11-23T01:15:55.8423022Z 772fa4efddc3: Pull complete
2022-11-23T01:16:24.2379216Z c36ac12376c5: Pull complete
2022-11-23T01:16:24.3380073Z 85cc9b957510: Pull complete
2022-11-23T01:16:24.4337697Z 9ab0826d88b5: Pull complete
2022-11-23T01:16:24.4435031Z Digest: sha256:a44ece0129de4f14f08fcb1423a34c97f2f88e2969bfc5ad6f33b15b4dfcfea3
2022-11-23T01:16:24.4463308Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:16:24.4491046Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:16:24.4591704Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2022-11-23T01:16:24.4592035Z with:
2022-11-23T01:16:24.4592278Z   driver-version: 515.76
2022-11-23T01:16:24.4592520Z env:
2022-11-23T01:16:24.4592744Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:16:24.4593002Z ##[endgroup]
2022-11-23T01:16:24.4626073Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-11-23T01:16:24.4626372Z with:
2022-11-23T01:16:24.4626610Z   timeout_minutes: 10
2022-11-23T01:16:24.4626859Z   max_attempts: 3
2022-11-23T01:16:24.4632805Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x

        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms

        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            sudo yum groupinstall -y "Development Tools"
            # ensure our kernel install is the same as our underlying kernel,
            # groupinstall "Development Tools" has a habit of mismatching kernel headers
            sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
            sudo modprobe backlight
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi

        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e
            nvidia-smi
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-11-23T01:16:24.4638926Z   retry_wait_seconds: 10
2022-11-23T01:16:24.4639210Z   polling_interval_seconds: 1
2022-11-23T01:16:24.4639486Z   warning_on_retry: true
2022-11-23T01:16:24.4639732Z   continue_on_error: false
2022-11-23T01:16:24.4639978Z env:
2022-11-23T01:16:24.4640283Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:16:24.4640539Z   DRIVER_VERSION: 515.76
2022-11-23T01:16:24.4640793Z ##[endgroup]
2022-11-23T01:16:24.5220272Z 
2022-11-23T01:16:24.5242435Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:16:24.5286579Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run ==
2022-11-23T01:16:24.5288666Z + sudo yum remove -y nvidia-driver-latest-dkms
2022-11-23T01:16:25.0762164Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:25.1362531Z No Match for argument: nvidia-driver-latest-dkms
2022-11-23T01:16:25.1741087Z No Packages marked for removal
2022-11-23T01:16:25.1899791Z + echo 'Before installing NVIDIA driver'
2022-11-23T01:16:25.1900085Z + lspci
2022-11-23T01:16:25.1900346Z Before installing NVIDIA driver
2022-11-23T01:16:25.2072383Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:16:25.2072799Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:16:25.2073217Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:16:25.2073615Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:16:25.2074217Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:16:25.2078151Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:16:25.2078817Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:25.2079272Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:25.2079704Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:16:25.2080024Z + lsmod
2022-11-23T01:16:25.2095091Z Module                  Size  Used by
2022-11-23T01:16:25.2095393Z nvidia_modeset       1142784  0
2022-11-23T01:16:25.2095665Z nvidia_uvm           1269760  0
2022-11-23T01:16:25.2095910Z veth                   16384  0
2022-11-23T01:16:25.2096414Z nvidia              40808448  2 nvidia_uvm,nvidia_modeset
2022-11-23T01:16:25.2096903Z drm                   425984  1 nvidia
2022-11-23T01:16:25.2097413Z i2c_core               77824  2 nvidia,drm
2022-11-23T01:16:25.2098032Z backlight              16384  1 nvidia_modeset
2022-11-23T01:16:25.2098605Z xt_conntrack           16384  1
2022-11-23T01:16:25.2100290Z ipt_MASQUERADE         16384  1
2022-11-23T01:16:25.2100636Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:16:25.2100965Z nf_conntrack_netlink    49152  0
2022-11-23T01:16:25.2101273Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:16:25.2101551Z xfrm_user              45056  1
2022-11-23T01:16:25.2101827Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:16:25.2102104Z xt_addrtype            16384  2
2022-11-23T01:16:25.2102360Z iptable_filter         16384  1
2022-11-23T01:16:25.2102629Z iptable_nat            16384  1
2022-11-23T01:16:25.2102898Z nf_conntrack_ipv4      16384  3
2022-11-23T01:16:25.2103179Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:16:25.2103488Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:16:25.2103821Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:16:25.2104300Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:16:25.2104691Z br_netfilter           24576  0
2022-11-23T01:16:25.2106380Z bridge                172032  1 br_netfilter
2022-11-23T01:16:25.2106685Z stp                    16384  1 bridge
2022-11-23T01:16:25.2106979Z llc                    16384  2 bridge,stp
2022-11-23T01:16:25.2107255Z overlay                86016  0
2022-11-23T01:16:25.2107501Z sunrpc                393216  1
2022-11-23T01:16:25.2107762Z dm_mirror              28672  0
2022-11-23T01:16:25.2108042Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:16:25.2108338Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:16:25.2108681Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:16:25.2108960Z dax                    69632  1 dm_mod
2022-11-23T01:16:25.2109212Z sb_edac                24576  0
2022-11-23T01:16:25.2109475Z crc32_pclmul           16384  0
2022-11-23T01:16:25.2109748Z ghash_clmulni_intel    16384  0
2022-11-23T01:16:25.2110066Z pcbc                   16384  0
2022-11-23T01:16:25.2110306Z ata_piix               36864  0
2022-11-23T01:16:25.2110572Z aesni_intel           188416  0
2022-11-23T01:16:25.2110848Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:16:25.2111120Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:16:25.2111395Z mousedev               24576  0
2022-11-23T01:16:25.2111666Z libata                266240  1 ata_piix
2022-11-23T01:16:25.2111962Z glue_helper            16384  1 aesni_intel
2022-11-23T01:16:25.2112223Z pcc_cpufreq            16384  0
2022-11-23T01:16:25.2112560Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:16:25.2112889Z scsi_mod              245760  1 libata
2022-11-23T01:16:25.2113138Z evdev                  20480  3
2022-11-23T01:16:25.2113575Z psmouse                32768  0
2022-11-23T01:16:25.2113832Z button                 16384  0
2022-11-23T01:16:25.2114065Z ena                   114688  0
2022-11-23T01:16:25.2114317Z xen_blkfront           49152  2
2022-11-23T01:16:25.2114578Z crc32c_intel           24576  0
2022-11-23T01:16:25.2114895Z autofs4                49152  2
2022-11-23T01:16:25.2115158Z + modinfo nvidia
2022-11-23T01:16:25.2115650Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2022-11-23T01:16:25.2115987Z firmware:       nvidia/515.76/gsp.bin
2022-11-23T01:16:25.2116331Z alias:          char-major-195-*
2022-11-23T01:16:25.2116601Z version:        515.76
2022-11-23T01:16:25.2116843Z supported:      external
2022-11-23T01:16:25.2117099Z license:        NVIDIA
2022-11-23T01:16:25.2117374Z srcversion:     51FD9DD90150B35351AFFBB
2022-11-23T01:16:25.2117672Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2022-11-23T01:16:25.2117989Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2022-11-23T01:16:25.2118305Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2022-11-23T01:16:25.2118645Z depends:        i2c-core,drm
2022-11-23T01:16:25.2118897Z retpoline:      Y
2022-11-23T01:16:25.2119144Z name:           nvidia
2022-11-23T01:16:25.2119549Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2022-11-23T01:16:25.2119920Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2022-11-23T01:16:25.2120324Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2022-11-23T01:16:25.2120693Z parm:           NVreg_ResmanDebugLevel:int
2022-11-23T01:16:25.2120979Z parm:           NVreg_RmLogonRC:int
2022-11-23T01:16:25.2121284Z parm:           NVreg_ModifyDeviceFiles:int
2022-11-23T01:16:25.2121591Z parm:           NVreg_DeviceFileUID:int
2022-11-23T01:16:25.2121874Z parm:           NVreg_DeviceFileGID:int
2022-11-23T01:16:25.2122177Z parm:           NVreg_DeviceFileMode:int
2022-11-23T01:16:25.2122542Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2022-11-23T01:16:25.2122907Z parm:           NVreg_UsePageAttributeTable:int
2022-11-23T01:16:25.2123228Z parm:           NVreg_EnablePCIeGen3:int
2022-11-23T01:16:25.2123520Z parm:           NVreg_EnableMSI:int
2022-11-23T01:16:25.2123812Z parm:           NVreg_TCEBypassMode:int
2022-11-23T01:16:25.2124116Z parm:           NVreg_EnableStreamMemOPs:int
2022-11-23T01:16:25.2124477Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2022-11-23T01:16:25.2124870Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2022-11-23T01:16:25.2125231Z parm:           NVreg_EnableS0ixPowerManagement:int
2022-11-23T01:16:25.2125637Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2022-11-23T01:16:25.2126034Z parm:           NVreg_DynamicPowerManagement:int
2022-11-23T01:16:25.2126430Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2022-11-23T01:16:25.2126830Z parm:           NVreg_EnableGpuFirmware:int
2022-11-23T01:16:25.2127161Z parm:           NVreg_EnableGpuFirmwareLogs:int
2022-11-23T01:16:25.2127515Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2022-11-23T01:16:25.2127887Z parm:           NVreg_EnableUserNUMAManagement:int
2022-11-23T01:16:25.2128220Z parm:           NVreg_MemoryPoolSize:int
2022-11-23T01:16:25.2128545Z parm:           NVreg_KMallocHeapMaxSize:int
2022-11-23T01:16:25.2128860Z parm:           NVreg_VMallocHeapMaxSize:int
2022-11-23T01:16:25.2129176Z parm:           NVreg_IgnoreMMIOCheck:int
2022-11-23T01:16:25.2129482Z parm:           NVreg_NvLinkDisable:int
2022-11-23T01:16:25.2129814Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2022-11-23T01:16:25.2130165Z parm:           NVreg_RegisterPCIDriver:int
2022-11-23T01:16:25.2130497Z parm:           NVreg_EnableDbgBreakpoint:int
2022-11-23T01:16:25.2130802Z parm:           NVreg_RegistryDwords:charp
2022-11-23T01:16:25.2131146Z parm:           NVreg_RegistryDwordsPerDevice:charp
2022-11-23T01:16:25.2132075Z parm:           NVreg_RmMsg:charp
2022-11-23T01:16:25.2132407Z parm:           NVreg_GpuBlacklist:charp
2022-11-23T01:16:25.2133021Z parm:           NVreg_TemporaryFilePath:charp
2022-11-23T01:16:25.2133342Z parm:           NVreg_ExcludedGpus:charp
2022-11-23T01:16:25.2133639Z parm:           NVreg_DmaRemapPeerMmio:int
2022-11-23T01:16:25.2133950Z parm:           rm_firmware_active:charp
2022-11-23T01:16:25.2134325Z + HAS_NVIDIA_DRIVER=0
2022-11-23T01:16:25.2134682Z ++ command -v nvidia-smi
2022-11-23T01:16:25.2134987Z + '[' -x /usr/bin/nvidia-smi ']'
2022-11-23T01:16:25.2135241Z + set +e
2022-11-23T01:16:25.2135642Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2022-11-23T01:16:28.9856704Z + INSTALLED_DRIVER_VERSION=515.76
2022-11-23T01:16:28.9857024Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:16:28.9858172Z + '[' 0 -ne 0 ']'
2022-11-23T01:16:28.9858496Z + '[' 515.76 '!=' 515.76 ']'
2022-11-23T01:16:28.9858759Z + HAS_NVIDIA_DRIVER=1
2022-11-23T01:16:28.9859260Z + echo 'NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation'
2022-11-23T01:16:28.9859629Z + set -e
2022-11-23T01:16:28.9859893Z + '[' 1 -eq 0 ']'
2022-11-23T01:16:28.9860142Z + sudo modprobe nvidia
2022-11-23T01:16:28.9860490Z NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation
2022-11-23T01:16:29.0001245Z + echo 'After installing NVIDIA driver'
2022-11-23T01:16:29.0001574Z + lspci
2022-11-23T01:16:29.0001817Z After installing NVIDIA driver
2022-11-23T01:16:29.0213812Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:16:29.0214259Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:16:29.0214882Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:16:29.0215271Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:16:29.0215641Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:16:29.0216029Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:16:29.0216448Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:29.0216884Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:29.0217310Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:16:29.0217640Z + lsmod
2022-11-23T01:16:29.0235438Z Module                  Size  Used by
2022-11-23T01:16:29.0235725Z nvidia_modeset       1142784  0
2022-11-23T01:16:29.0236011Z nvidia_uvm           1269760  0
2022-11-23T01:16:29.0236253Z veth                   16384  0
2022-11-23T01:16:29.0236551Z nvidia              40808448  27 nvidia_uvm,nvidia_modeset
2022-11-23T01:16:29.0237005Z drm                   425984  1 nvidia
2022-11-23T01:16:29.0237451Z i2c_core               77824  2 nvidia,drm
2022-11-23T01:16:29.0237742Z backlight              16384  1 nvidia_modeset
2022-11-23T01:16:29.0238023Z xt_conntrack           16384  1
2022-11-23T01:16:29.0238293Z ipt_MASQUERADE         16384  1
2022-11-23T01:16:29.0238590Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:16:29.0238896Z nf_conntrack_netlink    49152  0
2022-11-23T01:16:29.0239223Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:16:29.0239492Z xfrm_user              45056  1
2022-11-23T01:16:29.0239767Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:16:29.0240041Z xt_addrtype            16384  2
2022-11-23T01:16:29.0240307Z iptable_filter         16384  1
2022-11-23T01:16:29.0240603Z iptable_nat            16384  1
2022-11-23T01:16:29.0240867Z nf_conntrack_ipv4      16384  3
2022-11-23T01:16:29.0241167Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:16:29.0241456Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:16:29.0241780Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:16:29.0242252Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:16:29.0242862Z br_netfilter           24576  0
2022-11-23T01:16:29.0243139Z bridge                172032  1 br_netfilter
2022-11-23T01:16:29.0243417Z stp                    16384  1 bridge
2022-11-23T01:16:29.0243672Z llc                    16384  2 bridge,stp
2022-11-23T01:16:29.0244031Z overlay                86016  0
2022-11-23T01:16:29.0244300Z sunrpc                393216  1
2022-11-23T01:16:29.0244539Z dm_mirror              28672  0
2022-11-23T01:16:29.0244814Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:16:29.0245126Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:16:29.0245437Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:16:29.0245695Z dax                    69632  1 dm_mod
2022-11-23T01:16:29.0245957Z sb_edac                24576  0
2022-11-23T01:16:29.0246217Z crc32_pclmul           16384  0
2022-11-23T01:16:29.0246467Z ghash_clmulni_intel    16384  0
2022-11-23T01:16:29.0246734Z pcbc                   16384  0
2022-11-23T01:16:29.0246998Z ata_piix               36864  0
2022-11-23T01:16:29.0247240Z aesni_intel           188416  0
2022-11-23T01:16:29.0247510Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:16:29.0247797Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:16:29.0248053Z mousedev               24576  0
2022-11-23T01:16:29.0248321Z libata                266240  1 ata_piix
2022-11-23T01:16:29.0248610Z glue_helper            16384  1 aesni_intel
2022-11-23T01:16:29.0248870Z pcc_cpufreq            16384  0
2022-11-23T01:16:29.0249203Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:16:29.0249529Z scsi_mod              245760  1 libata
2022-11-23T01:16:29.0249771Z evdev                  20480  3
2022-11-23T01:16:29.0250026Z psmouse                32768  0
2022-11-23T01:16:29.0250280Z button                 16384  0
2022-11-23T01:16:29.0250528Z ena                   114688  0
2022-11-23T01:16:29.0250768Z xen_blkfront           49152  2
2022-11-23T01:16:29.0251026Z crc32c_intel           24576  0
2022-11-23T01:16:29.0251285Z autofs4                49152  2
2022-11-23T01:16:29.0252214Z + modinfo nvidia
2022-11-23T01:16:29.0252973Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2022-11-23T01:16:29.0253331Z firmware:       nvidia/515.76/gsp.bin
2022-11-23T01:16:29.0253660Z alias:          char-major-195-*
2022-11-23T01:16:29.0253931Z version:        515.76
2022-11-23T01:16:29.0254188Z supported:      external
2022-11-23T01:16:29.0254426Z license:        NVIDIA
2022-11-23T01:16:29.0254702Z srcversion:     51FD9DD90150B35351AFFBB
2022-11-23T01:16:29.0255016Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2022-11-23T01:16:29.0255332Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2022-11-23T01:16:29.0255628Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2022-11-23T01:16:29.0255963Z depends:        i2c-core,drm
2022-11-23T01:16:29.0256230Z retpoline:      Y
2022-11-23T01:16:29.0256457Z name:           nvidia
2022-11-23T01:16:29.0256863Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2022-11-23T01:16:29.0257243Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2022-11-23T01:16:29.0257626Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2022-11-23T01:16:29.0257998Z parm:           NVreg_ResmanDebugLevel:int
2022-11-23T01:16:29.0258298Z parm:           NVreg_RmLogonRC:int
2022-11-23T01:16:29.0258586Z parm:           NVreg_ModifyDeviceFiles:int
2022-11-23T01:16:29.0258895Z parm:           NVreg_DeviceFileUID:int
2022-11-23T01:16:29.0259192Z parm:           NVreg_DeviceFileGID:int
2022-11-23T01:16:29.0259480Z parm:           NVreg_DeviceFileMode:int
2022-11-23T01:16:29.0259843Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2022-11-23T01:16:29.0260218Z parm:           NVreg_UsePageAttributeTable:int
2022-11-23T01:16:29.0260541Z parm:           NVreg_EnablePCIeGen3:int
2022-11-23T01:16:29.0260819Z parm:           NVreg_EnableMSI:int
2022-11-23T01:16:29.0261239Z parm:           NVreg_TCEBypassMode:int
2022-11-23T01:16:29.0261559Z parm:           NVreg_EnableStreamMemOPs:int
2022-11-23T01:16:29.0261901Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2022-11-23T01:16:29.0262293Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2022-11-23T01:16:29.0262734Z parm:           NVreg_EnableS0ixPowerManagement:int
2022-11-23T01:16:29.0263137Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2022-11-23T01:16:29.0263533Z parm:           NVreg_DynamicPowerManagement:int
2022-11-23T01:16:29.0263945Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2022-11-23T01:16:29.0264322Z parm:           NVreg_EnableGpuFirmware:int
2022-11-23T01:16:29.0264657Z parm:           NVreg_EnableGpuFirmwareLogs:int
2022-11-23T01:16:29.0265016Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2022-11-23T01:16:29.0265384Z parm:           NVreg_EnableUserNUMAManagement:int
2022-11-23T01:16:29.0265710Z parm:           NVreg_MemoryPoolSize:int
2022-11-23T01:16:29.0266031Z parm:           NVreg_KMallocHeapMaxSize:int
2022-11-23T01:16:29.0266354Z parm:           NVreg_VMallocHeapMaxSize:int
2022-11-23T01:16:29.0266652Z parm:           NVreg_IgnoreMMIOCheck:int
2022-11-23T01:16:29.0266956Z parm:           NVreg_NvLinkDisable:int
2022-11-23T01:16:29.0267310Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2022-11-23T01:16:29.0267646Z parm:           NVreg_RegisterPCIDriver:int
2022-11-23T01:16:29.0267974Z parm:           NVreg_EnableDbgBreakpoint:int
2022-11-23T01:16:29.0268298Z parm:           NVreg_RegistryDwords:charp
2022-11-23T01:16:29.0268618Z parm:           NVreg_RegistryDwordsPerDevice:charp
2022-11-23T01:16:29.0268935Z parm:           NVreg_RmMsg:charp
2022-11-23T01:16:29.0269228Z parm:           NVreg_GpuBlacklist:charp
2022-11-23T01:16:29.0269533Z parm:           NVreg_TemporaryFilePath:charp
2022-11-23T01:16:29.0269853Z parm:           NVreg_ExcludedGpus:charp
2022-11-23T01:16:29.0270162Z parm:           NVreg_DmaRemapPeerMmio:int
2022-11-23T01:16:29.0270476Z parm:           rm_firmware_active:charp
2022-11-23T01:16:29.0270721Z + set +e
2022-11-23T01:16:29.0270988Z + nvidia-smi
2022-11-23T01:16:29.0448158Z Wed Nov 23 01:16:29 2022       
2022-11-23T01:16:29.0448618Z +-----------------------------------------------------------------------------+
2022-11-23T01:16:29.0449136Z | NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
2022-11-23T01:16:29.0449622Z |-------------------------------+----------------------+----------------------+
2022-11-23T01:16:29.0450119Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-11-23T01:16:29.0450623Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-11-23T01:16:29.0450990Z |                               |                      |               MIG M. |
2022-11-23T01:16:29.0451293Z |===============================+======================+======================|
2022-11-23T01:16:29.0496787Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |           8030816438 |
2022-11-23T01:16:29.0497158Z | N/A   28C    P0    38W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-11-23T01:16:29.0497487Z |                               |                      |                  N/A |
2022-11-23T01:16:29.0497961Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:16:29.0548438Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |          12884871354 |
2022-11-23T01:16:29.0548798Z | N/A   35C    P0    37W / 150W |      0MiB /  7680MiB |     66%      Default |
2022-11-23T01:16:29.0549128Z |                               |                      |                  N/A |
2022-11-23T01:16:29.0549576Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:16:29.0549954Z                                                                                
2022-11-23T01:16:29.0550397Z +-----------------------------------------------------------------------------+
2022-11-23T01:16:29.0550929Z | Processes:                                                                  |
2022-11-23T01:16:29.0551259Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-11-23T01:16:29.0551673Z |        ID   ID                                                   Usage      |
2022-11-23T01:16:29.0551994Z |=============================================================================|
2022-11-23T01:16:29.0560437Z |  No running processes found                                                 |
2022-11-23T01:16:29.0560911Z +-----------------------------------------------------------------------------+
2022-11-23T01:16:29.1076900Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:16:29.1077676Z + '[' 0 -eq 0 ']'
2022-11-23T01:16:29.1078083Z + echo 'INFO: Ignoring allowed status 0'
2022-11-23T01:16:29.1078389Z + set -e
2022-11-23T01:16:29.1078643Z INFO: Ignoring allowed status 0
2022-11-23T01:16:29.1082484Z == Installing nvidia container toolkit for amzn2 ==
2022-11-23T01:16:29.1086169Z + sudo yum install -y yum-utils
2022-11-23T01:16:29.6567524Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:31.0032067Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-11-23T01:16:31.0032780Z Nothing to do
2022-11-23T01:16:31.0823390Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:16:31.6561347Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:31.7002108Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:16:31.7003611Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:16:31.7004880Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:16:31.7154298Z + sudo yum install -y nvidia-docker2
2022-11-23T01:16:32.2689823Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:33.5837908Z Package nvidia-docker2-2.11.0-1.noarch already installed and latest version
2022-11-23T01:16:33.5838338Z Nothing to do
2022-11-23T01:16:33.6593032Z + sudo systemctl restart docker
2022-11-23T01:17:00.5671697Z Command completed after 1 attempt(s).
2022-11-23T01:17:00.5673974Z 
2022-11-23T01:17:00.5678851Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:17:00.5769830Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-11-23T01:17:00.5770460Z python3 -m pip install psutil==5.9.1
2022-11-23T01:17:00.5771035Z python3 -m pip install pynvml==11.4.1
2022-11-23T01:17:00.5772074Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-11-23T01:17:00.5772715Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2022-11-23T01:17:00.5788520Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:17:00.5788820Z env:
2022-11-23T01:17:00.5789065Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:00.5789320Z   GPU_FLAG: --gpus all
2022-11-23T01:17:00.5789569Z ##[endgroup]
2022-11-23T01:17:00.8845644Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:17:00.9080033Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2022-11-23T01:17:01.5058421Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:17:01.5290755Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2022-11-23T01:17:01.8248068Z Prepare all required actions
2022-11-23T01:17:01.8248448Z Getting action download info
2022-11-23T01:17:02.0008681Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2022-11-23T01:17:02.2080674Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7)
2022-11-23T01:17:02.3611086Z ##[group]Run ./.github/actions/download-build-artifacts
2022-11-23T01:17:02.3611397Z with:
2022-11-23T01:17:02.3612363Z   name: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:17:02.3612637Z env:
2022-11-23T01:17:02.3612882Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:02.3613153Z   GPU_FLAG: --gpus all
2022-11-23T01:17:02.3613386Z ##[endgroup]
2022-11-23T01:17:02.3643402Z ##[group]Run seemethere/download-artifact-s3@v4
2022-11-23T01:17:02.3643695Z with:
2022-11-23T01:17:02.3643961Z   name: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:17:02.3644268Z   s3-bucket: gha-artifacts
2022-11-23T01:17:02.3644579Z   region: us-east-1
2022-11-23T01:17:02.3644818Z env:
2022-11-23T01:17:02.3645061Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:02.3645314Z   GPU_FLAG: --gpus all
2022-11-23T01:17:02.3645566Z ##[endgroup]
2022-11-23T01:17:02.9391624Z Found 1 objects with prefix pytorch/pytorch/3528293554/linux-bionic-cuda11.7-py3.10-gcc7/
2022-11-23T01:17:02.9392268Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:17:09.2600502Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:17:09.2601066Z 
2022-11-23T01:17:09.2606101Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:17:09.2607361Z Artifact download has finished successfully
2022-11-23T01:17:09.2902241Z ##[group]Run unzip -o artifacts.zip
2022-11-23T01:17:09.2902570Z unzip -o artifacts.zip
2022-11-23T01:17:09.2916210Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:17:09.2916514Z env:
2022-11-23T01:17:09.2916759Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:09.2917023Z   GPU_FLAG: --gpus all
2022-11-23T01:17:09.2917278Z ##[endgroup]
2022-11-23T01:17:09.2960918Z Archive:  artifacts.zip
2022-11-23T01:17:09.2962876Z    creating: dist/
2022-11-23T01:17:11.3602843Z   inflating: dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl  
2022-11-23T01:17:11.3603265Z    creating: build/custom_test_artifacts/
2022-11-23T01:17:11.3603695Z    creating: build/custom_test_artifacts/custom-op-build/
2022-11-23T01:17:11.3604179Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-11-23T01:17:11.3610718Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:17:11.3611257Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-11-23T01:17:11.3612142Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:17:11.3612721Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:17:11.3613279Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:17:11.3615682Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:17:11.3616764Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:17:11.3617340Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:17:11.3617910Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:17:11.3620578Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:17:11.3621884Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:17:11.3623308Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:17:11.3624423Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:17:11.3625884Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:17:11.3627236Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:17:11.3627834Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:17:11.3628407Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:17:11.3682788Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:17:11.3683535Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:17:11.3684268Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:17:11.3685448Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:17:11.3686185Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:17:11.3686873Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:17:11.3687575Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:17:11.3688454Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:17:11.3689320Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:17:11.3731614Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:17:11.3773281Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:17:11.3774233Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:17:11.3775073Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:17:11.3775710Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:17:11.3776546Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:17:11.3777550Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:17:11.3778564Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:17:11.3780616Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:17:11.3854238Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:17:11.3927120Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:17:11.3928040Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:17:11.3928621Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-11-23T01:17:11.3929696Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-11-23T01:17:11.3930338Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:17:11.3930869Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-11-23T01:17:11.3931850Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-11-23T01:17:11.3932504Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-11-23T01:17:11.3933112Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-11-23T01:17:11.3933678Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-11-23T01:17:11.3934275Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:17:11.3934874Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-11-23T01:17:11.3935476Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-11-23T01:17:11.3936053Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-11-23T01:17:11.3936652Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-11-23T01:17:11.3957531Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-11-23T01:17:11.4071740Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-11-23T01:17:11.4072312Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-11-23T01:17:11.4072901Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-11-23T01:17:11.4073541Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-11-23T01:17:11.4074164Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-11-23T01:17:11.4074761Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-11-23T01:17:11.4075358Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:17:11.4076392Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-11-23T01:17:11.4077239Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-11-23T01:17:11.4077875Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-11-23T01:17:11.4078499Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-11-23T01:17:11.4099897Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-11-23T01:17:11.4182575Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-11-23T01:17:11.4183250Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:17:11.4183869Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:17:11.4184439Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-11-23T01:17:11.4185146Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-11-23T01:17:11.4186285Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:17:11.4186826Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-11-23T01:17:11.4189870Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-11-23T01:17:11.4190566Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-11-23T01:17:11.4191232Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-11-23T01:17:11.4284323Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-11-23T01:17:11.4347260Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-11-23T01:17:11.4347948Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-11-23T01:17:11.4348427Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-11-23T01:17:11.4355000Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:17:11.4355540Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-11-23T01:17:11.4356071Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:17:11.4356644Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:17:11.4357201Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:17:11.4359314Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:17:11.4360461Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:17:11.4361023Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:17:11.4361588Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:17:11.4364463Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:17:11.4365404Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:17:11.4367393Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:17:11.4368015Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:17:11.4369434Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:17:11.4370546Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:17:11.4371139Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:17:11.4372269Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:17:11.4426172Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:17:11.4426881Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:17:11.4427800Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:17:11.4428561Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:17:11.4429280Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:17:11.4429989Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:17:11.4430683Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:17:11.4431375Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:17:11.4432574Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:17:11.4475343Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:17:11.4515859Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:17:11.4516992Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:17:11.4517884Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:17:11.4518512Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:17:11.4519328Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:17:11.4520172Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:17:11.4521194Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:17:11.4523170Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:17:11.4596849Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:17:11.4670629Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:17:11.4671269Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:17:11.4671799Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-11-23T01:17:11.4672338Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-11-23T01:17:11.4672892Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:17:11.4673445Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-11-23T01:17:11.4674017Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-11-23T01:17:11.4674648Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-11-23T01:17:11.4675252Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-11-23T01:17:11.4675834Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-11-23T01:17:11.4676406Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-11-23T01:17:11.4677288Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-11-23T01:17:11.4677924Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-11-23T01:17:11.4678529Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-11-23T01:17:11.4679132Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-11-23T01:17:11.4699678Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-11-23T01:17:11.4762835Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-11-23T01:17:11.4763474Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:17:11.4764084Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:17:11.4764638Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-11-23T01:17:11.4765692Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-11-23T01:17:11.4767044Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:17:11.4767636Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-11-23T01:17:11.4770323Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-11-23T01:17:11.4771016Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-11-23T01:17:11.4772274Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-11-23T01:17:11.4821425Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-11-23T01:17:11.4821896Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-11-23T01:17:11.4822390Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-11-23T01:17:11.4828976Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:17:11.4829557Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-11-23T01:17:11.4830119Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:17:11.4830708Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:17:11.4831293Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:17:11.4833851Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:17:11.4835030Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:17:11.4835622Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:17:11.4836211Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:17:11.4838773Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:17:11.4839846Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:17:11.4841541Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:17:11.4842196Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:17:11.4844196Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:17:11.4845069Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:17:11.4845705Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:17:11.4846302Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:17:11.4900676Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:17:11.4901442Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:17:11.4902210Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:17:11.4902978Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:17:11.4903732Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:17:11.4904443Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:17:11.4905300Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:17:11.4906254Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:17:11.4907183Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:17:11.4949292Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:17:11.4990734Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:17:11.4991983Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:17:11.4992900Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:17:11.4993560Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:17:11.4994378Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:17:11.4995194Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:17:11.4996302Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:17:11.4998259Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:17:11.5071675Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:17:11.5144968Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:17:11.5145639Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:17:11.5146231Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-11-23T01:17:11.5146937Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-11-23T01:17:11.5147784Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:17:11.5148376Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-11-23T01:17:11.5148988Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-11-23T01:17:11.5149637Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-11-23T01:17:11.5150276Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-11-23T01:17:11.5150901Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-11-23T01:17:11.5151531Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:17:11.5152518Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-11-23T01:17:11.5153296Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-11-23T01:17:11.5153927Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-11-23T01:17:11.5154568Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-11-23T01:17:11.5159362Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-11-23T01:17:11.5306706Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-11-23T01:17:11.5307364Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-11-23T01:17:11.5307986Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-11-23T01:17:11.5308760Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-11-23T01:17:11.5309406Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-11-23T01:17:11.5310045Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-11-23T01:17:11.5310669Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:17:11.5311318Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-11-23T01:17:11.5311976Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-11-23T01:17:11.5312624Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-11-23T01:17:11.5313254Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-11-23T01:17:11.5334398Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-11-23T01:17:11.5392589Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-11-23T01:17:11.5393276Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:17:11.5393902Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:17:11.5394497Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-11-23T01:17:11.5395263Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-11-23T01:17:11.5396419Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:17:11.5396968Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-11-23T01:17:11.5400015Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-11-23T01:17:11.5400900Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-11-23T01:17:11.5401665Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-11-23T01:17:11.5520626Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-11-23T01:17:11.5566592Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-11-23T01:17:11.5566968Z    creating: build/lib/
2022-11-23T01:17:11.5567436Z   inflating: build/lib/libclog.a     
2022-11-23T01:17:11.5633979Z   inflating: build/lib/libgtest.a    
2022-11-23T01:17:11.5644231Z   inflating: build/lib/libpthreadpool.a  
2022-11-23T01:17:11.5750683Z   inflating: build/lib/libprotobuf-lite.a  
2022-11-23T01:17:11.5841756Z   inflating: build/lib/libbenchmark.a  
2022-11-23T01:17:11.5850928Z   inflating: build/lib/libittnotify.a  
2022-11-23T01:17:11.5883075Z   inflating: build/lib/libtensorpipe_uv.a  
2022-11-23T01:17:11.5959703Z   inflating: build/lib/libasmjit.a   
2022-11-23T01:17:11.6493254Z   inflating: build/lib/libprotobuf.a  
2022-11-23T01:17:11.6625752Z   inflating: build/lib/libgloo.a     
2022-11-23T01:17:11.6658268Z   inflating: build/lib/libfmt.a      
2022-11-23T01:17:11.6660056Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-11-23T01:17:11.6660677Z   inflating: build/lib/libfoxi_loader.a  
2022-11-23T01:17:11.6740267Z   inflating: build/lib/libc10.so     
2022-11-23T01:17:11.6741534Z   inflating: build/lib/libtorch_global_deps.so  
2022-11-23T01:17:11.6751576Z   inflating: build/lib/libcpuinfo.a  
2022-11-23T01:17:11.6760455Z   inflating: build/lib/libcpuinfo_internals.a  
2022-11-23T01:17:11.6762959Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-11-23T01:17:11.7333153Z   inflating: build/lib/libprotoc.a   
2022-11-23T01:17:11.7351902Z   inflating: build/lib/libgmock.a    
2022-11-23T01:17:11.7352564Z   inflating: build/lib/libgtest_main.a  
2022-11-23T01:17:11.7353530Z   inflating: build/lib/libbenchmark_main.a  
2022-11-23T01:17:11.7495575Z   inflating: build/lib/libXNNPACK.a  
2022-11-23T01:17:12.7275619Z   inflating: build/lib/libdnnl.a     
2022-11-23T01:17:12.7930657Z   inflating: build/lib/libtensorpipe.a  
2022-11-23T01:17:12.7983713Z   inflating: build/lib/libc10_cuda.so  
2022-11-23T01:17:12.7999392Z   inflating: build/lib/libqnnpack.a  
2022-11-23T01:17:12.7999956Z   inflating: build/lib/libgmock_main.a  
2022-11-23T01:17:12.9542160Z   inflating: build/lib/libfbgemm.a   
2022-11-23T01:17:12.9565108Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-11-23T01:17:13.0711835Z   inflating: build/lib/libdnnl_graph.a  
2022-11-23T01:17:13.1228137Z   inflating: build/lib/libkineto.a   
2022-11-23T01:17:13.1517733Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-11-23T01:17:13.1563289Z   inflating: build/lib/libcaffe2_protos.a  
2022-11-23T01:17:13.1611008Z   inflating: build/lib/libonnx_proto.a  
2022-11-23T01:17:13.1633589Z   inflating: build/lib/libnnpack.a   
2022-11-23T01:17:13.2311704Z   inflating: build/lib/libonnx.a     
2022-11-23T01:17:13.2745499Z   inflating: build/lib/libgloo_cuda.a  
2022-11-23T01:17:15.6375313Z   inflating: build/lib/libtorch_cpu.so  
2022-11-23T01:17:17.7608753Z   inflating: build/lib/libtorch_cuda.so  
2022-11-23T01:17:17.7609554Z   inflating: build/lib/libtorch.so   
2022-11-23T01:17:17.7612914Z   inflating: build/lib/libc10d_cuda_test.so  
2022-11-23T01:17:18.7486308Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-11-23T01:17:18.7510219Z   inflating: build/lib/libjitbackend_test.so  
2022-11-23T01:17:18.7570065Z   inflating: build/lib/libtorchbind_test.so  
2022-11-23T01:17:18.7601001Z   inflating: build/lib/libbackend_with_compiler.so  
2022-11-23T01:17:18.7605660Z   inflating: build/lib/libshm.so     
2022-11-23T01:17:18.9410948Z   inflating: build/lib/libtorch_python.so  
2022-11-23T01:17:18.9450508Z   inflating: build/lib/libnnapi_backend.so  
2022-11-23T01:17:18.9450826Z    creating: build/bin/
2022-11-23T01:17:18.9503429Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-11-23T01:17:18.9558521Z   inflating: build/bin/c10_DeviceGuard_test  
2022-11-23T01:17:18.9612091Z   inflating: build/bin/c10_Device_test  
2022-11-23T01:17:18.9663208Z   inflating: build/bin/c10_StreamGuard_test  
2022-11-23T01:17:18.9726019Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-11-23T01:17:18.9778715Z   inflating: build/bin/c10_SymInt_test  
2022-11-23T01:17:18.9838163Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-11-23T01:17:18.9898009Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-11-23T01:17:18.9958829Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-11-23T01:17:19.0009935Z   inflating: build/bin/c10_Array_test  
2022-11-23T01:17:19.0066711Z   inflating: build/bin/c10_Bitset_test  
2022-11-23T01:17:19.0121543Z   inflating: build/bin/c10_C++17_test  
2022-11-23T01:17:19.0173142Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-11-23T01:17:19.0225318Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-11-23T01:17:19.0278244Z   inflating: build/bin/c10_Half_test  
2022-11-23T01:17:19.0338691Z   inflating: build/bin/c10_LeftRight_test  
2022-11-23T01:17:19.0405681Z   inflating: build/bin/c10_Metaprogramming_test  
2022-11-23T01:17:19.0561520Z   inflating: build/bin/c10_SmallVectorTest  
2022-11-23T01:17:19.0615040Z   inflating: build/bin/c10_Synchronized_test  
2022-11-23T01:17:19.0676281Z   inflating: build/bin/c10_ThreadLocal_test  
2022-11-23T01:17:19.0732359Z   inflating: build/bin/c10_TypeIndex_test  
2022-11-23T01:17:19.0786245Z   inflating: build/bin/c10_TypeList_test  
2022-11-23T01:17:19.0837432Z   inflating: build/bin/c10_TypeTraits_test  
2022-11-23T01:17:19.0892787Z   inflating: build/bin/c10_accumulate_test  
2022-11-23T01:17:19.0952480Z   inflating: build/bin/c10_bfloat16_test  
2022-11-23T01:17:19.1009933Z   inflating: build/bin/c10_complex_math_test  
2022-11-23T01:17:19.1069349Z   inflating: build/bin/c10_complex_test  
2022-11-23T01:17:19.1187047Z   inflating: build/bin/c10_either_test  
2022-11-23T01:17:19.1243072Z   inflating: build/bin/c10_exception_test  
2022-11-23T01:17:19.1296589Z   inflating: build/bin/c10_flags_test  
2022-11-23T01:17:19.1479345Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-11-23T01:17:19.1533187Z   inflating: build/bin/c10_irange_test  
2022-11-23T01:17:19.1594858Z   inflating: build/bin/c10_logging_test  
2022-11-23T01:17:19.1674820Z   inflating: build/bin/c10_optional_test  
2022-11-23T01:17:19.1741250Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-11-23T01:17:19.1799378Z   inflating: build/bin/c10_registry_test  
2022-11-23T01:17:19.1862876Z   inflating: build/bin/c10_string_view_test  
2022-11-23T01:17:19.1917728Z   inflating: build/bin/c10_tempfile_test  
2022-11-23T01:17:19.1978357Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-11-23T01:17:19.2038379Z   inflating: build/bin/c10_typeid_test  
2022-11-23T01:17:19.2559935Z   inflating: build/bin/protoc-3.13.0.0  
2022-11-23T01:17:19.3081242Z   inflating: build/bin/protoc        
2022-11-23T01:17:19.3133026Z   inflating: build/bin/c10_cuda_CUDATest  
2022-11-23T01:17:19.3448982Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-11-23T01:17:19.3804252Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-11-23T01:17:19.3869022Z   inflating: build/bin/TCPStoreTest  
2022-11-23T01:17:19.3926136Z   inflating: build/bin/HashStoreTest  
2022-11-23T01:17:19.3984405Z   inflating: build/bin/FileStoreTest  
2022-11-23T01:17:19.4000135Z   inflating: build/bin/ProcessGroupMPITest  
2022-11-23T01:17:19.4003376Z   inflating: build/bin/example_allreduce  
2022-11-23T01:17:19.4059481Z   inflating: build/bin/Dimname_test  
2022-11-23T01:17:19.4138046Z   inflating: build/bin/Dict_test     
2022-11-23T01:17:19.4206297Z   inflating: build/bin/MaybeOwned_test  
2022-11-23T01:17:19.4267690Z   inflating: build/bin/NamedTensor_test  
2022-11-23T01:17:19.4330643Z   inflating: build/bin/apply_utils_test  
2022-11-23T01:17:19.4393682Z   inflating: build/bin/atest         
2022-11-23T01:17:19.4459712Z   inflating: build/bin/basic         
2022-11-23T01:17:19.4517287Z   inflating: build/bin/broadcast_test  
2022-11-23T01:17:19.4579666Z   inflating: build/bin/cpu_generator_test  
2022-11-23T01:17:19.4635457Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-11-23T01:17:19.4688762Z   inflating: build/bin/dispatch_key_set_test  
2022-11-23T01:17:19.4783917Z   inflating: build/bin/cpu_rng_test  
2022-11-23T01:17:19.4836844Z   inflating: build/bin/dlconvertor_test  
2022-11-23T01:17:19.4899265Z   inflating: build/bin/extension_backend_test  
2022-11-23T01:17:19.4958730Z   inflating: build/bin/half_test     
2022-11-23T01:17:19.5060551Z   inflating: build/bin/ivalue_test   
2022-11-23T01:17:19.5113306Z   inflating: build/bin/lazy_tensor_test  
2022-11-23T01:17:19.5170480Z   inflating: build/bin/memory_format_test  
2022-11-23T01:17:19.5228314Z   inflating: build/bin/math_kernel_test  
2022-11-23T01:17:19.5285243Z   inflating: build/bin/memory_overlapping_test  
2022-11-23T01:17:19.5345395Z   inflating: build/bin/native_test   
2022-11-23T01:17:19.5401020Z   inflating: build/bin/mobile_memory_cleanup  
2022-11-23T01:17:19.5455503Z   inflating: build/bin/operator_name_test  
2022-11-23T01:17:19.5508919Z   inflating: build/bin/operators_test  
2022-11-23T01:17:19.5565164Z   inflating: build/bin/packedtensoraccessor_test  
2022-11-23T01:17:19.5635407Z   inflating: build/bin/pow_test      
2022-11-23T01:17:19.5696818Z   inflating: build/bin/quantized_test  
2022-11-23T01:17:19.5749545Z   inflating: build/bin/reduce_ops_test  
2022-11-23T01:17:19.5803541Z   inflating: build/bin/reportMemoryUsage_test  
2022-11-23T01:17:19.5864077Z   inflating: build/bin/scalar_tensor_test  
2022-11-23T01:17:19.5925357Z   inflating: build/bin/scalar_test   
2022-11-23T01:17:19.5981251Z   inflating: build/bin/stride_properties_test  
2022-11-23T01:17:19.6065899Z   inflating: build/bin/tensor_iterator_test  
2022-11-23T01:17:19.6068553Z   inflating: build/bin/thread_init_test  
2022-11-23T01:17:19.6128245Z   inflating: build/bin/type_ptr_test  
2022-11-23T01:17:19.6187922Z   inflating: build/bin/test_parallel  
2022-11-23T01:17:19.6240830Z   inflating: build/bin/variant_test  
2022-11-23T01:17:19.6305760Z   inflating: build/bin/type_test     
2022-11-23T01:17:19.6361342Z   inflating: build/bin/undefined_tensor_test  
2022-11-23T01:17:19.6362516Z   inflating: build/bin/verify_api_visibility  
2022-11-23T01:17:19.6437301Z   inflating: build/bin/vmap_test     
2022-11-23T01:17:19.6492004Z   inflating: build/bin/weakref_test  
2022-11-23T01:17:19.6546335Z   inflating: build/bin/wrapdim_test  
2022-11-23T01:17:19.6610446Z   inflating: build/bin/IListRef_test  
2022-11-23T01:17:19.6728660Z   inflating: build/bin/List_test     
2022-11-23T01:17:19.6781109Z   inflating: build/bin/xla_tensor_test  
2022-11-23T01:17:19.6912356Z   inflating: build/bin/kernel_function_legacy_test  
2022-11-23T01:17:19.7016181Z   inflating: build/bin/kernel_function_test  
2022-11-23T01:17:19.7085974Z   inflating: build/bin/KernelFunction_test  
2022-11-23T01:17:19.7224672Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-11-23T01:17:19.7336276Z   inflating: build/bin/kernel_lambda_test  
2022-11-23T01:17:19.7400277Z   inflating: build/bin/kernel_stackbased_test  
2022-11-23T01:17:19.7503620Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-11-23T01:17:19.7557313Z   inflating: build/bin/CppSignature_test  
2022-11-23T01:17:19.7608650Z   inflating: build/bin/op_allowlist_test  
2022-11-23T01:17:19.7665227Z   inflating: build/bin/inline_container_test  
2022-11-23T01:17:19.7725455Z   inflating: build/bin/backend_fallback_test  
2022-11-23T01:17:19.8039375Z   inflating: build/bin/op_registration_test  
2022-11-23T01:17:19.8095203Z   inflating: build/bin/cuda_apply_test  
2022-11-23T01:17:19.8159312Z   inflating: build/bin/cuda_atomic_ops_test  
2022-11-23T01:17:19.8216536Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-11-23T01:17:19.8289736Z   inflating: build/bin/cuda_complex_math_test  
2022-11-23T01:17:19.8342103Z   inflating: build/bin/cuda_device_test  
2022-11-23T01:17:19.8405905Z   inflating: build/bin/cuda_complex_test  
2022-11-23T01:17:19.8469363Z   inflating: build/bin/cuda_cub_test  
2022-11-23T01:17:19.8522582Z   inflating: build/bin/cuda_dlconvertor_test  
2022-11-23T01:17:19.8576623Z   inflating: build/bin/cuda_integer_divider_test  
2022-11-23T01:17:19.8648510Z   inflating: build/bin/cuda_distributions_test  
2022-11-23T01:17:19.8711276Z   inflating: build/bin/cuda_generator_test  
2022-11-23T01:17:19.8763759Z   inflating: build/bin/cuda_half_test  
2022-11-23T01:17:19.8816045Z   inflating: build/bin/cuda_optional_test  
2022-11-23T01:17:19.8871978Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-11-23T01:17:19.8936876Z   inflating: build/bin/cuda_stream_test  
2022-11-23T01:17:19.8991510Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-11-23T01:17:19.9043520Z   inflating: build/bin/cuda_cudnn_test  
2022-11-23T01:17:19.9099466Z   inflating: build/bin/cuda_vectorized_test  
2022-11-23T01:17:19.9117127Z   inflating: build/bin/tutorial_tensorexpr  
2022-11-23T01:17:19.9187215Z   inflating: build/bin/ProcessGroupGlooTest  
2022-11-23T01:17:19.9249447Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-11-23T01:17:19.9315446Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-11-23T01:17:19.9378116Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-11-23T01:17:19.9434630Z   inflating: build/bin/ProcessGroupUCCTest  
2022-11-23T01:17:19.9492636Z   inflating: build/bin/test_dist_autograd  
2022-11-23T01:17:19.9567388Z   inflating: build/bin/test_cpp_rpc  
2022-11-23T01:17:19.9570024Z   inflating: build/bin/parallel_benchmark  
2022-11-23T01:17:19.9644211Z   inflating: build/bin/test_mobile_nnc  
2022-11-23T01:17:19.9655720Z   inflating: build/bin/aot_model_compiler_test  
2022-11-23T01:17:20.0570488Z   inflating: build/bin/test_tensorexpr  
2022-11-23T01:17:20.0957284Z   inflating: build/bin/test_lazy     
2022-11-23T01:17:20.0962806Z   inflating: build/bin/torch_shm_manager  
2022-11-23T01:17:20.2289830Z   inflating: build/bin/test_api      
2022-11-23T01:17:20.3500196Z   inflating: build/bin/test_jit      
2022-11-23T01:17:20.3503527Z   inflating: .pytorch-test-times.json  
2022-11-23T01:17:20.3533407Z ##[group]Run df -H
2022-11-23T01:17:20.3533668Z df -H
2022-11-23T01:17:20.3546858Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:17:20.3547145Z env:
2022-11-23T01:17:20.3547391Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:20.3547678Z   GPU_FLAG: --gpus all
2022-11-23T01:17:20.3547916Z ##[endgroup]
2022-11-23T01:17:20.3587364Z Filesystem      Size  Used Avail Use% Mounted on
2022-11-23T01:17:20.3587713Z devtmpfs        129G     0  129G   0% /dev
2022-11-23T01:17:20.3588007Z tmpfs           129G  6.3M  129G   1% /dev/shm
2022-11-23T01:17:20.3588287Z tmpfs           129G  553k  129G   1% /run
2022-11-23T01:17:20.3588579Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-11-23T01:17:20.3589149Z /dev/xvda1      162G   29G  134G  18% /
2022-11-23T01:17:20.3612566Z ##[group]Run .github/scripts/parse_ref.py
2022-11-23T01:17:20.3612919Z .github/scripts/parse_ref.py
2022-11-23T01:17:20.3625009Z shell: /usr/bin/bash -e {0}
2022-11-23T01:17:20.3625267Z env:
2022-11-23T01:17:20.3625494Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:20.3625769Z   GPU_FLAG: --gpus all
2022-11-23T01:17:20.3626026Z ##[endgroup]
2022-11-23T01:17:20.3920222Z ##[group]Run set -x
2022-11-23T01:17:20.3920611Z set -x
2022-11-23T01:17:20.3920847Z 
2022-11-23T01:17:20.3921107Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-11-23T01:17:20.3921462Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-11-23T01:17:20.3921818Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-11-23T01:17:20.3922136Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-11-23T01:17:20.3922412Z else
2022-11-23T01:17:20.3922694Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:17:20.3922976Z fi
2022-11-23T01:17:20.3923181Z 
2022-11-23T01:17:20.3923507Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-11-23T01:17:20.3923829Z 
2022-11-23T01:17:20.3924116Z # sanitize the input commit message and PR body here:
2022-11-23T01:17:20.3924413Z #
2022-11-23T01:17:20.3924798Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-11-23T01:17:20.3925296Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-11-23T01:17:20.3925725Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-11-23T01:17:20.3926047Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-11-23T01:17:20.3926295Z 
2022-11-23T01:17:20.3926656Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-11-23T01:17:20.3927040Z # wreak havoc internally
2022-11-23T01:17:20.3927365Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-11-23T01:17:20.3927683Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-11-23T01:17:20.3927946Z 
2022-11-23T01:17:20.3928258Z # detached container should get cleaned up by teardown_ec2_linux
2022-11-23T01:17:20.3928652Z # TODO: Stop building test binaries as part of the build phase
2022-11-23T01:17:20.3929031Z # Used for GPU_FLAG since that doesn't play nice
2022-11-23T01:17:20.3929364Z # shellcheck disable=SC2086,SC2090
2022-11-23T01:17:20.3929670Z container_name=$(docker run \
2022-11-23T01:17:20.3929932Z   ${GPU_FLAG:-} \
2022-11-23T01:17:20.3930209Z   -e BUILD_ENVIRONMENT \
2022-11-23T01:17:20.3930618Z   -e PR_NUMBER \
2022-11-23T01:17:20.3930872Z   -e GITHUB_ACTIONS \
2022-11-23T01:17:20.3931138Z   -e BASE_SHA \
2022-11-23T01:17:20.3931391Z   -e BRANCH \
2022-11-23T01:17:20.3932133Z   -e SHA1 \
2022-11-23T01:17:20.3932442Z   -e AWS_DEFAULT_REGION \
2022-11-23T01:17:20.3932718Z   -e IN_WHEEL_TEST \
2022-11-23T01:17:20.3932968Z   -e SHARD_NUMBER \
2022-11-23T01:17:20.3933232Z   -e TEST_CONFIG \
2022-11-23T01:17:20.3933500Z   -e NUM_TEST_SHARDS \
2022-11-23T01:17:20.3933748Z   -e PR_BODY \
2022-11-23T01:17:20.3934016Z   -e COMMIT_MESSAGES \
2022-11-23T01:17:20.3934311Z   -e PYTORCH_RETRY_TEST_CASES \
2022-11-23T01:17:20.3934612Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-11-23T01:17:20.3934902Z   -e PR_LABELS \
2022-11-23T01:17:20.3935193Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-11-23T01:17:20.3935486Z   -e SCCACHE_BUCKET \
2022-11-23T01:17:20.3935756Z   -e SCCACHE_S3_KEY_PREFIX \
2022-11-23T01:17:20.3936030Z   -e XLA_CUDA \
2022-11-23T01:17:20.3936316Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-11-23T01:17:20.3936622Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2022-11-23T01:17:20.3936955Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2022-11-23T01:17:20.3937311Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-11-23T01:17:20.3937621Z   --ulimit stack=10485760:83886080 \
2022-11-23T01:17:20.3937938Z   --security-opt seccomp=unconfined \
2022-11-23T01:17:20.3938363Z   --cap-add=SYS_PTRACE \
2022-11-23T01:17:20.3938656Z   --ipc=host \
2022-11-23T01:17:20.3938909Z   --shm-size="${SHM_SIZE}" \
2022-11-23T01:17:20.3939171Z   --tty \
2022-11-23T01:17:20.3939413Z   --detach \
2022-11-23T01:17:20.3939667Z   --name="${container_name}" \
2022-11-23T01:17:20.3939948Z   --user jenkins \
2022-11-23T01:17:20.3940276Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-11-23T01:17:20.3940604Z   -w /var/lib/jenkins/workspace \
2022-11-23T01:17:20.3940889Z   "${DOCKER_IMAGE}"
2022-11-23T01:17:20.3941131Z )
2022-11-23T01:17:20.3941417Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2022-11-23T01:17:20.3941872Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2022-11-23T01:17:20.3954010Z shell: /usr/bin/bash -e {0}
2022-11-23T01:17:20.3954265Z env:
2022-11-23T01:17:20.3954514Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:20.3954766Z   GPU_FLAG: --gpus all
2022-11-23T01:17:20.3955099Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:17:20.3955412Z   PR_NUMBER: 
2022-11-23T01:17:20.3955633Z   BRANCH: master
2022-11-23T01:17:20.3955910Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:20.3956231Z   BASE_SHA: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:20.3956520Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T01:17:20.3956810Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T01:17:20.3957092Z   TEST_CONFIG: distributed
2022-11-23T01:17:20.3957343Z   SHARD_NUMBER: 2
2022-11-23T01:17:20.3957568Z   NUM_TEST_SHARDS: 3
2022-11-23T01:17:20.3957807Z   PR_BODY: 
2022-11-23T01:17:20.3958111Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-11-23T01:17:20.3958424Z   SCCACHE_S3_KEY_PREFIX: trunk
2022-11-23T01:17:20.3958680Z   SHM_SIZE: 2g
2022-11-23T01:17:20.3959174Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:17:20.3959631Z   XLA_CUDA: 
2022-11-23T01:17:20.3959985Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:17:20.3960369Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2022-11-23T01:17:20.3960657Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2022-11-23T01:17:20.3961050Z ##[endgroup]
2022-11-23T01:17:20.3988934Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-11-23T01:17:20.3989447Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *onnx* ]]
2022-11-23T01:17:20.3989790Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:17:20.3992680Z ++ git cherry -v origin/master
2022-11-23T01:17:20.4009208Z + COMMIT_MESSAGES=
2022-11-23T01:17:20.4009472Z + COMMIT_MESSAGES=
2022-11-23T01:17:20.4009696Z + PR_BODY=
2022-11-23T01:17:20.4010130Z + export COMMIT_MESSAGES=
2022-11-23T01:17:20.4010418Z + COMMIT_MESSAGES=
2022-11-23T01:17:20.4010660Z + export PR_BODY=
2022-11-23T01:17:20.4010903Z + PR_BODY=
2022-11-23T01:17:20.4020313Z +++ nproc --ignore=2
2022-11-23T01:17:20.4032871Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528293554 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:17:34.4059238Z + container_name=9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T01:17:34.4059810Z + echo DOCKER_CONTAINER_ID=9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T01:17:34.4063833Z ++ echo dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:17:34.4065466Z + docker exec -t 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c sh -c 'pip install dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2022-11-23T01:17:34.9710784Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:17:35.9267728Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (1.11.1)
2022-11-23T01:17:35.9271126Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (4.4.0)
2022-11-23T01:17:35.9275561Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3)
2022-11-23T01:17:35.9291825Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (3.3.0)
2022-11-23T01:17:35.9370871Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==1.14.0a0+git1cfd385) (1.21.2)
2022-11-23T01:17:35.9585852Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1)
2022-11-23T01:17:36.8619342Z Installing collected packages: torch
2022-11-23T01:17:46.5853432Z Successfully installed torch-1.14.0a0+git1cfd385
2022-11-23T01:17:46.6507748Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-11-23T01:17:46.6728750Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2022-11-23T01:17:46.6729774Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T01:17:46.6731056Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2022-11-23T01:17:46.6732697Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2022-11-23T01:17:46.6733202Z + BUILD_DIR=build
2022-11-23T01:17:46.6733482Z + BUILD_RENAMED_DIR=build_renamed
2022-11-23T01:17:46.6733746Z + BUILD_BIN_DIR=build/bin
2022-11-23T01:17:46.6734027Z + export VALGRIND=ON
2022-11-23T01:17:46.6734601Z + VALGRIND=ON
2022-11-23T01:17:46.6735323Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *clang9* ]]
2022-11-23T01:17:46.6735816Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *bazel* ]]
2022-11-23T01:17:46.6736173Z ++ realpath build/custom_test_artifacts
2022-11-23T01:17:46.6742888Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-11-23T01:17:46.6747442Z ++ dirname .jenkins/pytorch/test.sh
2022-11-23T01:17:46.6755276Z + source .jenkins/pytorch/common.sh
2022-11-23T01:17:46.6759097Z +++ dirname .jenkins/pytorch/common.sh
2022-11-23T01:17:46.6769896Z ++ source .jenkins/pytorch/common_utils.sh
2022-11-23T01:17:46.6772302Z +++ declare -f -t trap_add
2022-11-23T01:17:46.6778640Z ++ set -ex
2022-11-23T01:17:46.6779026Z ++ [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:17:46.6779332Z ++ BUILD_TEST_LIBTORCH=0
2022-11-23T01:17:46.6780384Z + echo 'Environment variables'
2022-11-23T01:17:46.6780669Z Environment variables
2022-11-23T01:17:46.6780909Z + env
2022-11-23T01:17:46.6789055Z SHARD_NUMBER=2
2022-11-23T01:17:46.6789548Z NV_LIBCUBLAS_DEV_VERSION=11.10.1.25-1
2022-11-23T01:17:46.6789905Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-7
2022-11-23T01:17:46.6790274Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-11-23T01:17:46.6790707Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.13.4-1+cuda11.7
2022-11-23T01:17:46.6791000Z UCC_HOME=/usr
2022-11-23T01:17:46.6791550Z BUILD_ENVIRONMENT=linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:17:46.6791910Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2022-11-23T01:17:46.6794748Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-7=11.7.3.21-1
2022-11-23T01:17:46.6795481Z INSTALLED_DB=yes
2022-11-23T01:17:46.6795833Z HOSTNAME=9de1665d20b3
2022-11-23T01:17:46.6796101Z GITHUB_REF_NAME=master
2022-11-23T01:17:46.6796404Z GITHUB_API_URL=https://api.github.com
2022-11-23T01:17:46.6796716Z OPENSSL_DIR=/opt/openssl
2022-11-23T01:17:46.6797033Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2022-11-23T01:17:46.6797962Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_0cc87b9b-2306-4393-ac1e-0f1e56ddeee4
2022-11-23T01:17:46.6798409Z CUDA_PATH=/usr/local/cuda
2022-11-23T01:17:46.6798915Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-11-23T01:17:46.6799314Z GITHUB_RUN_ATTEMPT=1
2022-11-23T01:17:46.6799566Z TEST_CONFIG=distributed
2022-11-23T01:17:46.6799871Z NV_LIBNPP_VERSION=11.7.3.21-1
2022-11-23T01:17:46.6800255Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-7=11.7.50-1
2022-11-23T01:17:46.6800563Z GITHUB_REPOSITORY_OWNER=pytorch
2022-11-23T01:17:46.6800849Z GITHUB_ACTIONS=true
2022-11-23T01:17:46.6801117Z NVIDIA_VISIBLE_DEVICES=all
2022-11-23T01:17:46.6803881Z NV_NVPROF_VERSION=11.7.50-1
2022-11-23T01:17:46.6804236Z NV_LIBCUSPARSE_VERSION=11.7.3.50-1
2022-11-23T01:17:46.6804536Z CI=true
2022-11-23T01:17:46.6804813Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-11-23T01:17:46.6805240Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-7=11.10.1.25-1
2022-11-23T01:17:46.6805576Z BRANCH=master
2022-11-23T01:17:46.6805837Z GITHUB_HEAD_REF=
2022-11-23T01:17:46.6806147Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2022-11-23T01:17:46.6806494Z GITHUB_ACTOR=pytorchmergebot
2022-11-23T01:17:46.6806845Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2022-11-23T01:17:46.6807148Z GITHUB_ACTION_REF=
2022-11-23T01:17:46.6807461Z NCCL_VERSION=2.13.4-1
2022-11-23T01:17:46.6807734Z GITHUB_ACTION=__self
2022-11-23T01:17:46.6808130Z VALGRIND=ON
2022-11-23T01:17:46.6808569Z GITHUB_REF_PROTECTED=true
2022-11-23T01:17:46.6809060Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:17:46.6809459Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2022-11-23T01:17:46.6812466Z ***
2022-11-23T01:17:46.6812731Z INSTALLED_VISION=yes
2022-11-23T01:17:46.6812988Z NVARCH=x86_64
2022-11-23T01:17:46.6813310Z NV_LIBCUSPARSE_DEV_VERSION=11.7.3.50-1
2022-11-23T01:17:46.6813597Z HOME=/var/lib/jenkins
2022-11-23T01:17:46.6814144Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_0cc87b9b-2306-4393-ac1e-0f1e56ddeee4
2022-11-23T01:17:46.6814724Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-11-23T01:17:46.6815012Z GITHUB_ACTION_REPOSITORY=
2022-11-23T01:17:46.6815290Z GITHUB_REF_TYPE=branch
2022-11-23T01:17:46.6815594Z NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1
2022-11-23T01:17:46.6815885Z GITHUB_RETENTION_DAYS=90
2022-11-23T01:17:46.6816274Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-11-23T01:17:46.6816694Z NV_LIBNCCL_PACKAGE=libnccl2=2.13.4-1+cuda11.7
2022-11-23T01:17:46.6817689Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_0cc87b9b-2306-4393-ac1e-0f1e56ddeee4
2022-11-23T01:17:46.6818427Z DEBIAN_FRONTEND=noninteractive
2022-11-23T01:17:46.6819185Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2022-11-23T01:17:46.6819788Z GITHUB_REF=refs/heads/master
2022-11-23T01:17:46.6820263Z NV_CUDA_LIB_VERSION=11.7.0-1
2022-11-23T01:17:46.6820579Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:46.6820894Z INSTALLED_PROTOBUF=yes
2022-11-23T01:17:46.6821145Z GITHUB_RUN_ID=3528293554
2022-11-23T01:17:46.6821494Z NV_LIBNPP_PACKAGE=libnpp-11-7=11.7.3.21-1
2022-11-23T01:17:46.6821947Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2022-11-23T01:17:46.6822367Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2022-11-23T01:17:46.6822694Z NV_NVTX_VERSION=11.7.50-1
2022-11-23T01:17:46.6823002Z GITHUB_SERVER_URL=https://github.com
2022-11-23T01:17:46.6823265Z MAX_JOBS=30
2022-11-23T01:17:46.6823553Z NV_LIBCUBLAS_VERSION=11.10.1.25-1
2022-11-23T01:17:46.6823930Z NV_LIBCUBLAS_PACKAGE=libcublas-11-7=11.10.1.25-1
2022-11-23T01:17:46.6824538Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-11-23T01:17:46.6824918Z UCX_HOME=/usr
2022-11-23T01:17:46.6825179Z PYTORCH_RETRY_TEST_CASES=1
2022-11-23T01:17:46.6825498Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-11-23T01:17:46.6825857Z BASE_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:46.6826204Z NV_CUDA_CUDART_DEV_VERSION=11.7.60-1
2022-11-23T01:17:46.6826480Z PR_BODY=
2022-11-23T01:17:46.6826698Z GITHUB_BASE_REF=
2022-11-23T01:17:46.6826937Z TERM=xterm
2022-11-23T01:17:46.6827165Z XLA_CUDA=
2022-11-23T01:17:46.6827417Z NV_NVML_DEV_VERSION=11.7.50-1
2022-11-23T01:17:46.6827697Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-11-23T01:17:46.6827958Z CUDA_VERSION=11.7.0
2022-11-23T01:17:46.6828280Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-7
2022-11-23T01:17:46.6828583Z OPENSSL_ROOT_DIR=/opt/openssl
2022-11-23T01:17:46.6829132Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_0cc87b9b-2306-4393-ac1e-0f1e56ddeee4
2022-11-23T01:17:46.6829515Z GITHUB_JOB=test
2022-11-23T01:17:46.6829775Z SCCACHE_S3_KEY_PREFIX=trunk
2022-11-23T01:17:46.6830040Z COMMIT_MESSAGES=
2022-11-23T01:17:46.6830318Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2022-11-23T01:17:46.6830610Z NUM_TEST_SHARDS=3
2022-11-23T01:17:46.6830847Z PR_NUMBER=
2022-11-23T01:17:46.6831352Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_0cc87b9b-2306-4393-ac1e-0f1e56ddeee4
2022-11-23T01:17:46.6831741Z SHLVL=1
2022-11-23T01:17:46.6832084Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-7
2022-11-23T01:17:46.6832401Z GITHUB_REPOSITORY=pytorch/pytorch
2022-11-23T01:17:46.6833528Z NVIDIA_REQUIRE_CUDA=cuda>=11.7 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511
2022-11-23T01:17:46.6834784Z NV_LIBNPP_DEV_VERSION=11.7.3.21-1
2022-11-23T01:17:46.6835077Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:46.6835368Z GITHUB_EVENT_NAME=push
2022-11-23T01:17:46.6835676Z NV_CUDA_CUDART_VERSION=11.7.60-1
2022-11-23T01:17:46.6836015Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-11-23T01:17:46.6836310Z GITHUB_RUN_NUMBER=18336
2022-11-23T01:17:46.6836572Z GITHUB_WORKFLOW=trunk
2022-11-23T01:17:46.6836974Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:17:46.6837434Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.13.4-1
2022-11-23T01:17:46.6837876Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:17:46.6838248Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2022-11-23T01:17:46.6838510Z _=/usr/bin/env
2022-11-23T01:17:46.6838797Z + echo 'Testing pytorch'
2022-11-23T01:17:46.6839179Z Testing pytorch
2022-11-23T01:17:46.6839477Z + export LANG=C.UTF-8
2022-11-23T01:17:46.6839819Z + LANG=C.UTF-8
2022-11-23T01:17:46.6840166Z + PR_NUMBER=
2022-11-23T01:17:46.6840520Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-11-23T01:17:46.6840853Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-11-23T01:17:46.6841335Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:17:46.6841728Z + [[ distributed == \s\l\o\w ]]
2022-11-23T01:17:46.6842262Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *slow-gradcheck* ]]
2022-11-23T01:17:46.6842738Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *cuda* ]]
2022-11-23T01:17:46.6843153Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-11-23T01:17:46.6843607Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-11-23T01:17:46.6843940Z + [[ distributed == *crossref* ]]
2022-11-23T01:17:46.6844353Z + [[ distributed == *dynamo* ]]
2022-11-23T01:17:46.6844768Z + [[ distributed == *inductor* ]]
2022-11-23T01:17:46.6845181Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:17:46.6845683Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *-bazel-* ]]
2022-11-23T01:17:46.6846190Z + pip_install --user ninja==1.10.2
2022-11-23T01:17:46.6846607Z + pip install --progress-bar off --user ninja==1.10.2
2022-11-23T01:17:47.2274247Z Collecting ninja==1.10.2
2022-11-23T01:17:47.2485118Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-11-23T01:17:48.1181292Z Installing collected packages: ninja
2022-11-23T01:17:48.1281866Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-11-23T01:17:48.1283036Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T01:17:48.1342175Z Successfully installed ninja-1.10.2
2022-11-23T01:17:48.2054712Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:17:48.2055987Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:17:48.2057257Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *asan* ]]
2022-11-23T01:17:48.2058344Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-tsan* ]]
2022-11-23T01:17:48.2058999Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-11-23T01:17:48.2059348Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-11-23T01:17:48.2064264Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *tbb* ]]
2022-11-23T01:17:48.2079176Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *libtorch* ]]
2022-11-23T01:17:48.2080289Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-bazel-* ]]
2022-11-23T01:17:48.2080772Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-tsan* ]]
2022-11-23T01:17:48.2082673Z + cd test
2022-11-23T01:17:48.2083603Z + python -c 'import torch; print(torch.__config__.show())'
2022-11-23T01:17:49.8646130Z PyTorch built with:
2022-11-23T01:17:49.8647050Z   - GCC 7.5
2022-11-23T01:17:49.8648234Z   - C++ Version: 201402
2022-11-23T01:17:49.8649301Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:17:49.8650122Z   - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:17:49.8650595Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:17:49.8697438Z   - LAPACK is enabled (usually provided by MKL)
2022-11-23T01:17:49.8697868Z   - NNPACK is enabled
2022-11-23T01:17:49.8698168Z   - CPU capability usage: AVX2
2022-11-23T01:17:49.8698479Z   - CUDA Runtime 11.7
2022-11-23T01:17:49.8698886Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-11-23T01:17:49.8699218Z   - CuDNN 8.5
2022-11-23T01:17:49.8699489Z   - Magma 2.6.1
2022-11-23T01:17:49.8702694Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-11-23T01:17:49.8704903Z 
2022-11-23T01:17:50.1034233Z + cd test
2022-11-23T01:17:50.1034876Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-11-23T01:17:51.6997968Z ATen/Parallel:
2022-11-23T01:17:51.7013577Z 	at::get_num_threads() : 16
2022-11-23T01:17:51.7013918Z 	at::get_num_interop_threads() : 16
2022-11-23T01:17:51.7014225Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:17:51.7014493Z 	omp_get_max_threads() : 16
2022-11-23T01:17:51.7015142Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:17:51.7015556Z 	mkl_get_max_threads() : 16
2022-11-23T01:17:51.7015995Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:17:51.7016358Z std::thread::hardware_concurrency() : 32
2022-11-23T01:17:51.7016656Z Environment variables:
2022-11-23T01:17:51.7016912Z 	OMP_NUM_THREADS : [not set]
2022-11-23T01:17:51.7017184Z 	MKL_NUM_THREADS : [not set]
2022-11-23T01:17:51.7017474Z ATen parallel backend: OpenMP
2022-11-23T01:17:51.7017653Z 
2022-11-23T01:17:51.9334296Z + [[ distributed == *backward* ]]
2022-11-23T01:17:51.9334623Z + [[ distributed == *xla* ]]
2022-11-23T01:17:51.9334922Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-11-23T01:17:51.9335459Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *libtorch* ]]
2022-11-23T01:17:51.9335790Z + [[ distributed == distributed ]]
2022-11-23T01:17:51.9336068Z + install_filelock
2022-11-23T01:17:51.9336328Z + pip_install filelock
2022-11-23T01:17:51.9336691Z + pip install --progress-bar off filelock
2022-11-23T01:17:52.4590515Z Collecting filelock
2022-11-23T01:17:52.4811873Z   Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
2022-11-23T01:17:53.3486505Z Installing collected packages: filelock
2022-11-23T01:17:53.3884372Z Successfully installed filelock-3.8.0
2022-11-23T01:17:53.4581409Z + install_triton
2022-11-23T01:17:53.4581676Z + local commit
2022-11-23T01:17:53.4581940Z + [[ distributed == *rocm* ]]
2022-11-23T01:17:53.4585339Z ++ get_pinned_commit triton
2022-11-23T01:17:53.4585652Z ++ cat .github/ci_commit_pins/triton.txt
2022-11-23T01:17:53.4600931Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:17:53.4601591Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:17:53.4602303Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:17:53.9478191Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:17:53.9484830Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-sp_z73yx
2022-11-23T01:17:53.9506952Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-sp_z73yx
2022-11-23T01:17:54.7436692Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2022-11-23T01:17:54.7459043Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:17:55.1527103Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:17:55.3575339Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:17:55.3578165Z   Running command git submodule update --init --recursive -q
2022-11-23T01:17:55.9739288Z   Preparing metadata (setup.py) ... [?25l- done
2022-11-23T01:17:56.1918646Z [?25hCollecting cmake
2022-11-23T01:17:56.2182573Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2022-11-23T01:17:56.5833247Z Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (3.8.0)
2022-11-23T01:17:56.5836263Z Requirement already satisfied: torch in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385)
2022-11-23T01:17:56.6091460Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (1.11.1)
2022-11-23T01:17:56.6096567Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (4.4.0)
2022-11-23T01:17:56.6101059Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (2.6.3)
2022-11-23T01:17:56.6313169Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2022-11-23T01:17:56.6384745Z Building wheels for collected packages: triton
2022-11-23T01:18:49.5378571Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | done
2022-11-23T01:18:49.5861177Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp310-cp310-linux_x86_64.whl size=15377935 sha256=42e10cfb16031f9cc89a776504abcf40ef9a3a5f4c38ee0ebf453e2f59df05ba
2022-11-23T01:18:49.5862584Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3f/1d/23/1c2bc47d618a44f9c949aea4b7e355e737a1f1ed208f009295
2022-11-23T01:18:49.5882701Z Successfully built triton
2022-11-23T01:18:50.4504332Z Installing collected packages: cmake, triton
2022-11-23T01:18:52.3439921Z Successfully installed cmake-3.25.0 triton-2.0.0
2022-11-23T01:18:52.4592403Z + pip_install --user jinja2
2022-11-23T01:18:52.4592867Z + pip install --progress-bar off --user jinja2
2022-11-23T01:18:53.4487133Z Collecting jinja2
2022-11-23T01:18:53.4700138Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2022-11-23T01:18:53.6950932Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2) (2.1.1)
2022-11-23T01:18:54.5990537Z Installing collected packages: jinja2
2022-11-23T01:18:54.7066813Z Successfully installed jinja2-3.1.2
2022-11-23T01:18:54.7732772Z + test_distributed
2022-11-23T01:18:54.7733239Z + echo 'Testing distributed python tests'
2022-11-23T01:18:54.7733868Z Testing distributed python tests
2022-11-23T01:18:54.7734310Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose
2022-11-23T01:18:57.0110492Z Ignoring disabled issues:  []
2022-11-23T01:18:57.0507628Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T01:18:57.0508212Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T01:18:57.0514529Z Found test time stats from artifacts
2022-11-23T01:18:57.0530437Z Selected tests:
2022-11-23T01:18:57.0531131Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:18:57.0531844Z  distributed/test_distributed_spawn
2022-11-23T01:18:57.0532161Z  distributed/rpc/test_tensorpipe_agent
2022-11-23T01:18:57.0532506Z  distributed/pipeline/sync/test_transparency
2022-11-23T01:18:57.0532841Z  distributed/pipeline/sync/test_pipe
2022-11-23T01:18:57.0533785Z  distributed/pipeline/sync/test_inplace
2022-11-23T01:18:57.0534100Z  distributed/pipeline/sync/test_copy
2022-11-23T01:18:57.0534419Z  distributed/pipeline/sync/test_balance
2022-11-23T01:18:57.0534760Z  distributed/pipeline/sync/skip/test_stash_pop
2022-11-23T01:18:57.0535107Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-11-23T01:18:57.0538319Z  distributed/optim/test_apply_optimizer_in_backward
2022-11-23T01:18:57.0538678Z  distributed/elastic/events/lib_test
2022-11-23T01:18:57.0538996Z  distributed/_shard/test_sharder
2022-11-23T01:18:57.0539306Z  distributed/nn/jit/test_instantiator
2022-11-23T01:18:57.0539822Z  distributed/checkpoint/test_planner
2022-11-23T01:18:57.0540150Z  distributed/fsdp/test_utils
2022-11-23T01:18:57.0544948Z  distributed/test_multi_threaded_pg
2022-11-23T01:18:57.0545526Z  distributed/elastic/utils/util_test
2022-11-23T01:18:57.0546231Z  distributed/rpc/test_share_memory
2022-11-23T01:18:57.0546912Z  distributed/elastic/utils/distributed_test
2022-11-23T01:18:57.0547492Z  distributed/fsdp/test_fsdp_fx
2022-11-23T01:18:57.0548107Z  distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T01:18:57.0548918Z  distributed/_tensor/parallel/test_2d_parallel
2022-11-23T01:18:57.0549523Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T01:18:57.0549885Z  distributed/_composable/test_replicate
2022-11-23T01:18:57.0550229Z  distributed/fsdp/test_distributed_checkpoint
2022-11-23T01:18:57.0550587Z  distributed/elastic/timer/local_timer_example
2022-11-23T01:18:57.0550933Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-11-23T01:18:57.0551289Z  distributed/fsdp/test_fsdp_apply
2022-11-23T01:18:57.0551594Z  distributed/_shard/test_partial_tensor
2022-11-23T01:18:57.0551879Z  distributed/_tensor/test_api
2022-11-23T01:18:57.0552183Z  distributed/test_c10d_object_collectives
2022-11-23T01:18:57.0552504Z  distributed/_tensor/test_tp_sharding_ops
2022-11-23T01:18:57.0552828Z  distributed/checkpoint/test_file_system_checkpoint
2022-11-23T01:18:57.0553169Z  distributed/checkpoint/test_checkpoint
2022-11-23T01:18:57.0553480Z  distributed/_tensor/test_redistribute
2022-11-23T01:18:57.0553766Z  distributed/fsdp/test_fsdp_meta
2022-11-23T01:18:57.0554084Z  distributed/fsdp/test_fsdp_flatten_params
2022-11-23T01:18:57.0554425Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-11-23T01:18:57.0554732Z  distributed/algorithms/test_join
2022-11-23T01:18:57.0555026Z  distributed/_tensor/test_dtensor
2022-11-23T01:18:57.0555316Z  distributed/fsdp/test_fsdp_misc
2022-11-23T01:18:57.0555597Z  distributed/_tensor/test_tensor_ops
2022-11-23T01:18:57.0555914Z  distributed/fsdp/test_fsdp_checkpoint
2022-11-23T01:18:57.0556248Z  distributed/optim/test_zero_redundancy_optimizer
2022-11-23T01:18:57.0556583Z  distributed/fsdp/test_fsdp_optim_state
2022-11-23T01:18:57.0556878Z  distributed/fsdp/test_fsdp_state_dict
2022-11-23T01:18:57.0557168Z  distributed/test_c10d_nccl
2022-11-23T01:18:57.0557478Z Prioritized test from test file changes.
2022-11-23T01:18:57.0557762Z reordering tests for PR:
2022-11-23T01:18:57.0558220Z prioritized: []
2022-11-23T01:18:57.0562638Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/rpc/test_tensorpipe_agent', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/optim/test_apply_optimizer_in_backward', 'distributed/elastic/events/lib_test', 'distributed/_shard/test_sharder', 'distributed/nn/jit/test_instantiator', 'distributed/checkpoint/test_planner', 'distributed/fsdp/test_utils', 'distributed/test_multi_threaded_pg', 'distributed/elastic/utils/util_test', 'distributed/rpc/test_share_memory', 'distributed/elastic/utils/distributed_test', 'distributed/fsdp/test_fsdp_fx', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/_tensor/parallel/test_2d_parallel', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_composable/test_replicate', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/elastic/timer/local_timer_example', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/fsdp/test_fsdp_apply', 'distributed/_shard/test_partial_tensor', 'distributed/_tensor/test_api', 'distributed/test_c10d_object_collectives', 'distributed/_tensor/test_tp_sharding_ops', 'distributed/checkpoint/test_file_system_checkpoint', 'distributed/checkpoint/test_checkpoint', 'distributed/_tensor/test_redistribute', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_flatten_params', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/algorithms/test_join', 'distributed/_tensor/test_dtensor', 'distributed/fsdp/test_fsdp_misc', 'distributed/_tensor/test_tensor_ops', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/fsdp/test_fsdp_optim_state', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/test_c10d_nccl']
2022-11-23T01:18:57.0565508Z 
2022-11-23T01:18:57.0566054Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T01:18:57.0768095Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T01:18:57.0975625Z parallel (file granularity) tests:
2022-11-23T01:18:57.0975930Z  
2022-11-23T01:18:57.0976311Z serial (file granularity) tests:
2022-11-23T01:18:57.0976964Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:18:57.0977762Z  distributed/test_distributed_spawn
2022-11-23T01:18:57.0978297Z  distributed/rpc/test_tensorpipe_agent
2022-11-23T01:18:57.0978640Z  distributed/pipeline/sync/test_transparency
2022-11-23T01:18:57.0978976Z  distributed/pipeline/sync/test_pipe
2022-11-23T01:18:57.0979290Z  distributed/pipeline/sync/test_inplace
2022-11-23T01:18:57.0979588Z  distributed/pipeline/sync/test_copy
2022-11-23T01:18:57.0979899Z  distributed/pipeline/sync/test_balance
2022-11-23T01:18:57.0980230Z  distributed/pipeline/sync/skip/test_stash_pop
2022-11-23T01:18:57.0980577Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-11-23T01:18:57.0980943Z  distributed/optim/test_apply_optimizer_in_backward
2022-11-23T01:18:57.0981268Z  distributed/elastic/events/lib_test
2022-11-23T01:18:57.0981548Z  distributed/_shard/test_sharder
2022-11-23T01:18:57.0981849Z  distributed/nn/jit/test_instantiator
2022-11-23T01:18:57.0982158Z  distributed/checkpoint/test_planner
2022-11-23T01:18:57.0982431Z  distributed/fsdp/test_utils
2022-11-23T01:18:57.0982720Z  distributed/test_multi_threaded_pg
2022-11-23T01:18:57.0983017Z  distributed/elastic/utils/util_test
2022-11-23T01:18:57.0983299Z  distributed/rpc/test_share_memory
2022-11-23T01:18:57.0983786Z  distributed/elastic/utils/distributed_test
2022-11-23T01:18:57.0984097Z  distributed/fsdp/test_fsdp_fx
2022-11-23T01:18:57.0984399Z  distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T01:18:57.0984742Z  distributed/_tensor/parallel/test_2d_parallel
2022-11-23T01:18:57.0985106Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T01:18:57.0985455Z  distributed/_composable/test_replicate
2022-11-23T01:18:57.0985770Z  distributed/fsdp/test_distributed_checkpoint
2022-11-23T01:18:57.0986107Z  distributed/elastic/timer/local_timer_example
2022-11-23T01:18:57.0986475Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-11-23T01:18:57.0986796Z  distributed/fsdp/test_fsdp_apply
2022-11-23T01:18:57.0987102Z  distributed/_shard/test_partial_tensor
2022-11-23T01:18:57.0987400Z  distributed/_tensor/test_api
2022-11-23T01:18:57.0987686Z  distributed/test_c10d_object_collectives
2022-11-23T01:18:57.0988003Z  distributed/_tensor/test_tp_sharding_ops
2022-11-23T01:18:57.0988347Z  distributed/checkpoint/test_file_system_checkpoint
2022-11-23T01:18:57.0988664Z  distributed/checkpoint/test_checkpoint
2022-11-23T01:18:57.0988974Z  distributed/_tensor/test_redistribute
2022-11-23T01:18:57.0989275Z  distributed/fsdp/test_fsdp_meta
2022-11-23T01:18:57.0989568Z  distributed/fsdp/test_fsdp_flatten_params
2022-11-23T01:18:57.0989904Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-11-23T01:18:57.0990219Z  distributed/algorithms/test_join
2022-11-23T01:18:57.0990494Z  distributed/_tensor/test_dtensor
2022-11-23T01:18:57.0990787Z  distributed/fsdp/test_fsdp_misc
2022-11-23T01:18:57.0991175Z  distributed/_tensor/test_tensor_ops
2022-11-23T01:18:57.0991505Z  distributed/fsdp/test_fsdp_checkpoint
2022-11-23T01:18:57.0991824Z  distributed/optim/test_zero_redundancy_optimizer
2022-11-23T01:18:57.0992161Z  distributed/fsdp/test_fsdp_optim_state
2022-11-23T01:18:57.0992473Z  distributed/fsdp/test_fsdp_state_dict
2022-11-23T01:18:57.0992748Z  distributed/test_c10d_nccl
2022-11-23T01:18:59.2537217Z Ignoring disabled issues:  []
2022-11-23T01:18:59.2981371Z Ignoring disabled issues:  []
2022-11-23T01:18:59.7231367Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:18:59.722624]
2022-11-23T01:18:59.7250553Z /usr/bin/mpiexec
2022-11-23T01:18:59.7251174Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:18:59.7252350Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2022-11-23T01:18:59.7252765Z Shard 2: test should be run in 1
2022-11-23T01:18:59.7253161Z Running distributed tests for the nccl backend with env init_method in shard 2 of 3
2022-11-23T01:18:59.7255926Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:18:59.725222]
2022-11-23T01:19:36.7769492Z 
2022-11-23T01:19:36.7770091Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:19:36.7779197Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_wm7ckpt_)
2022-11-23T01:19:36.7780388Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:19:36.7781265Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:36.7781750Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:36.7782318Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:36.7782949Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:36.7783306Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:36.7783693Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:36.7784067Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:19:36.7784740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7785206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7785800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7786283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7786666Z 
2022-11-23T01:19:36.7786845Z Running tests...
2022-11-23T01:19:36.7787300Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7787910Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:36.7788473Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-11-23T01:19:36.7788770Z 
2022-11-23T01:19:36.7789044Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7789374Z Ran 1 test in 0.001s
2022-11-23T01:19:36.7789540Z 
2022-11-23T01:19:36.7789649Z OK (skipped=1)
2022-11-23T01:19:36.7789805Z 
2022-11-23T01:19:36.7789912Z Generating XML reports...
2022-11-23T01:19:36.7790717Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011903.xml
2022-11-23T01:19:36.7791508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7791961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7792611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7793073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7793306Z 
2022-11-23T01:19:36.7793418Z Running tests...
2022-11-23T01:19:36.7793832Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7794421Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:36.7795002Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-11-23T01:19:36.7795295Z 
2022-11-23T01:19:36.7795561Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7795900Z Ran 1 test in 0.001s
2022-11-23T01:19:36.7796062Z 
2022-11-23T01:19:36.7796154Z OK (skipped=1)
2022-11-23T01:19:36.7796308Z 
2022-11-23T01:19:36.7796435Z Generating XML reports...
2022-11-23T01:19:36.7797102Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011905.xml
2022-11-23T01:19:36.7797858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7798299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7798878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7799356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7799589Z 
2022-11-23T01:19:36.7799680Z Running tests...
2022-11-23T01:19:36.7800086Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7800687Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:36.7801232Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:19:36.7801773Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 977
2022-11-23T01:19:36.7802220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 978
2022-11-23T01:19:36.7802836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7803275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7803856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7804330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7804910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7805338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7805921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7806388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7806811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:36.7807291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:36.7807778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:36.7808339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:36.7809003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:36.7809545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:36.7810202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:36.7810736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:36.7811368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:19:36.7812778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:19:36.7813181Z ok (5.402s)
2022-11-23T01:19:36.7813338Z 
2022-11-23T01:19:36.7813607Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7813918Z Ran 1 test in 5.402s
2022-11-23T01:19:36.7814082Z 
2022-11-23T01:19:36.7814179Z OK
2022-11-23T01:19:36.7814315Z 
2022-11-23T01:19:36.7814441Z Generating XML reports...
2022-11-23T01:19:36.7815086Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011907.xml
2022-11-23T01:19:36.7815855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7816308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7816887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7817344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7817576Z 
2022-11-23T01:19:36.7817690Z Running tests...
2022-11-23T01:19:36.7818097Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7818680Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:36.7819222Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:19:36.7819823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1096
2022-11-23T01:19:36.7820273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1097
2022-11-23T01:19:36.7820871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7821325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7821903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7822364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7822949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7823392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7823964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7824422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7824869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:36.7825363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:36.7825851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:36.7826394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:36.7827081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:36.7827617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:36.7828252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:36.7828791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:36.7829437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:19:36.7830122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:19:36.7830499Z ok (5.461s)
2022-11-23T01:19:36.7830650Z 
2022-11-23T01:19:36.7830921Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7831249Z Ran 1 test in 5.461s
2022-11-23T01:19:36.7831412Z 
2022-11-23T01:19:36.7831506Z OK
2022-11-23T01:19:36.7831623Z 
2022-11-23T01:19:36.7831748Z Generating XML reports...
2022-11-23T01:19:36.7832409Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011915.xml
2022-11-23T01:19:36.7833166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7833599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7834176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7834648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7834878Z 
2022-11-23T01:19:36.7834988Z Running tests...
2022-11-23T01:19:36.7835377Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7835980Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:36.7836540Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:19:36.7837004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1215
2022-11-23T01:19:36.7837527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1216
2022-11-23T01:19:36.7838137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7838587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7839149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7839624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7840202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7840631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7841202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7841669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7842104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:36.7842578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:36.7843065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:36.7843590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:36.7844305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:36.7844836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:36.7845495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:36.7846029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:36.7846678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:19:36.7847344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:19:36.7847737Z ok (5.307s)
2022-11-23T01:19:36.7847887Z 
2022-11-23T01:19:36.7848152Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7848485Z Ran 1 test in 5.307s
2022-11-23T01:19:36.7848632Z 
2022-11-23T01:19:36.7848726Z OK
2022-11-23T01:19:36.7848861Z 
2022-11-23T01:19:36.7848985Z Generating XML reports...
2022-11-23T01:19:36.7849643Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011923.xml
2022-11-23T01:19:36.7850440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7850901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7851989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7852574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7852791Z 
2022-11-23T01:19:36.7852903Z Running tests...
2022-11-23T01:19:36.7853317Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7853931Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:36.7854470Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:19:36.7854952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1334
2022-11-23T01:19:36.7855523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1335
2022-11-23T01:19:36.7856141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7856576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7857154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7857624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7858189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:36.7858638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:36.7859212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:36.7859677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:36.7860102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:36.7860596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:36.7861083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:36.7861565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:36.7862282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:36.7862995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:36.7863535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:36.7864011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:36.7864670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:19:36.7865360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:19:36.7865757Z ok (5.351s)
2022-11-23T01:19:36.7865909Z 
2022-11-23T01:19:36.7866156Z ----------------------------------------------------------------------
2022-11-23T01:19:36.7866489Z Ran 1 test in 5.351s
2022-11-23T01:19:36.7866655Z 
2022-11-23T01:19:36.7866751Z OK
2022-11-23T01:19:36.7866885Z 
2022-11-23T01:19:36.7867010Z Generating XML reports...
2022-11-23T01:19:36.7867654Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011930.xml
2022-11-23T01:19:36.7868059Z 
2022-11-23T01:19:36.7868394Z ##[endgroup]
2022-11-23T01:19:36.7869112Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_wm7ckpt_)
2022-11-23T01:19:36.7869534Z 
2022-11-23T01:19:36.7869750Z Running distributed tests for the nccl backend with file init_method in shard 2 of 3
2022-11-23T01:19:36.7870540Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:19:36.778564]
2022-11-23T01:20:13.6388243Z 
2022-11-23T01:20:13.6389182Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:20:13.6390281Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_rku_v4_r)
2022-11-23T01:20:13.6391414Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:20:13.6392595Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:20:13.6392952Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:20:13.6393331Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:20:13.6394221Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:20:13.6394667Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:20:13.6395063Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:20:13.6395405Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:20:13.6396114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6396579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6397153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6397906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6398149Z 
2022-11-23T01:20:13.6398265Z Running tests...
2022-11-23T01:20:13.6398708Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6399500Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:20:13.6400108Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-11-23T01:20:13.6400406Z 
2022-11-23T01:20:13.6400677Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6400998Z Ran 1 test in 0.001s
2022-11-23T01:20:13.6401164Z 
2022-11-23T01:20:13.6406383Z OK (skipped=1)
2022-11-23T01:20:13.6406559Z 
2022-11-23T01:20:13.6406691Z Generating XML reports...
2022-11-23T01:20:13.6407721Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011940.xml
2022-11-23T01:20:13.6408843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6409314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6410211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6410711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6411014Z 
2022-11-23T01:20:13.6411215Z Running tests...
2022-11-23T01:20:13.6412532Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6413705Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:20:13.6414815Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-11-23T01:20:13.6415379Z 
2022-11-23T01:20:13.6415887Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6416525Z Ran 1 test in 0.001s
2022-11-23T01:20:13.6416839Z 
2022-11-23T01:20:13.6417061Z OK (skipped=1)
2022-11-23T01:20:13.6417381Z 
2022-11-23T01:20:13.6417600Z Generating XML reports...
2022-11-23T01:20:13.6418379Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011942.xml
2022-11-23T01:20:13.6419126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6419807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6420401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6420880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6421117Z 
2022-11-23T01:20:13.6421209Z Running tests...
2022-11-23T01:20:13.6421613Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6422220Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:20:13.6422756Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:20:13.6423238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1552
2022-11-23T01:20:13.6423690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1553
2022-11-23T01:20:13.6424303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6424748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6425333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6425806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6426391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6426937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6427543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6428015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6428440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:20:13.6428945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:20:13.6429439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:20:13.6429924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:20:13.6430570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:20:13.6431114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:20:13.6431767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:20:13.6432313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:20:13.6432943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:20:13.6433626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:20:13.6434024Z ok (5.413s)
2022-11-23T01:20:13.6434173Z 
2022-11-23T01:20:13.6434422Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6434758Z Ran 1 test in 5.413s
2022-11-23T01:20:13.6434921Z 
2022-11-23T01:20:13.6435017Z OK
2022-11-23T01:20:13.6435151Z 
2022-11-23T01:20:13.6435276Z Generating XML reports...
2022-11-23T01:20:13.6435926Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011944.xml
2022-11-23T01:20:13.6436689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6437143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6437785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6438263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6438494Z 
2022-11-23T01:20:13.6438606Z Running tests...
2022-11-23T01:20:13.6439015Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6439603Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:20:13.6440154Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:20:13.6440630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1671
2022-11-23T01:20:13.6441061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1672
2022-11-23T01:20:13.6441676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6442135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6442714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6443169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6443808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6444262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6444891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6445379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6445803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:20:13.6446276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:20:13.6446767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:20:13.6447249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:20:13.6447920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:20:13.6448461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:20:13.6449112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:20:13.6449633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:20:13.6450281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:20:13.6450968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:20:13.6451369Z ok (5.331s)
2022-11-23T01:20:13.6452246Z 
2022-11-23T01:20:13.6452549Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6452888Z Ran 1 test in 5.331s
2022-11-23T01:20:13.6453054Z 
2022-11-23T01:20:13.6453150Z OK
2022-11-23T01:20:13.6453286Z 
2022-11-23T01:20:13.6453397Z Generating XML reports...
2022-11-23T01:20:13.6454069Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011952.xml
2022-11-23T01:20:13.6454832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6455286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6455846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6456452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6456687Z 
2022-11-23T01:20:13.6456799Z Running tests...
2022-11-23T01:20:13.6457191Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6457802Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:20:13.6458365Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:20:13.6458859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1790
2022-11-23T01:20:13.6459295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1791
2022-11-23T01:20:13.6459901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6460360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6460937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6461395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6461976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6462426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6463057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6463548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6463991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:20:13.6464474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:20:13.6464953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:20:13.6465447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:20:13.6466116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:20:13.6466635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:20:13.6467290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:20:13.6467822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:20:13.6468470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:20:13.6469141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:20:13.6469548Z ok (5.327s)
2022-11-23T01:20:13.6469699Z 
2022-11-23T01:20:13.6469966Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6470297Z Ran 1 test in 5.327s
2022-11-23T01:20:13.6470441Z 
2022-11-23T01:20:13.6470539Z OK
2022-11-23T01:20:13.6470672Z 
2022-11-23T01:20:13.6470796Z Generating XML reports...
2022-11-23T01:20:13.6471464Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011959.xml
2022-11-23T01:20:13.6472202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6472656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6473238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6473789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6474021Z 
2022-11-23T01:20:13.6474114Z Running tests...
2022-11-23T01:20:13.6474523Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6475126Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-11-23T01:20:13.6475668Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:20:13.6476156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1909
2022-11-23T01:20:13.6476607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1910
2022-11-23T01:20:13.6477214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6477644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6478228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6478694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6479272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:20:13.6479701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:20:13.6480322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:20:13.6480803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:20:13.6481227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:20:13.6481703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:20:13.6482194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:20:13.6482694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:20:13.6483341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:20:13.6483880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:20:13.6484532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:20:13.6485065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:20:13.6485691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:20:13.6486371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:20:13.6486767Z ok (5.352s)
2022-11-23T01:20:13.6486915Z 
2022-11-23T01:20:13.6487165Z ----------------------------------------------------------------------
2022-11-23T01:20:13.6487500Z Ran 1 test in 5.353s
2022-11-23T01:20:13.6487664Z 
2022-11-23T01:20:13.6487760Z OK
2022-11-23T01:20:13.6487895Z 
2022-11-23T01:20:13.6488022Z Generating XML reports...
2022-11-23T01:20:13.6488668Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012007.xml
2022-11-23T01:20:13.6489074Z 
2022-11-23T01:20:13.6489529Z ##[endgroup]
2022-11-23T01:20:13.6490252Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_rku_v4_r)
2022-11-23T01:20:13.6490692Z 
2022-11-23T01:20:13.6490822Z Shard 2: gloo should be run in 1
2022-11-23T01:20:13.6491169Z Shard 2: ucc should be run in 1
2022-11-23T01:20:13.6492294Z Running distributed/test_distributed_spawn ... [2022-11-23 01:20:13.640172]
2022-11-23T01:20:13.6492630Z /usr/bin/mpiexec
2022-11-23T01:20:13.6492992Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:20:13.6493552Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 3}
2022-11-23T01:20:13.6493949Z Shard 2: test should be run in 1
2022-11-23T01:20:13.6494305Z Running distributed tests for the nccl backend with env init_method in shard 2 of 3
2022-11-23T01:20:13.6495059Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:20:13.642184]
2022-11-23T01:44:15.2036359Z 
2022-11-23T01:44:15.2038905Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:44:15.2039854Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_tbddl287)
2022-11-23T01:44:15.2043202Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:44:15.2083252Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T01:44:15.2139427Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2140359Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2141156Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2141883Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2142669Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2143537Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2144366Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2145190Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2146123Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2147089Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2148086Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2148988Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2149931Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2151057Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2151895Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2152740Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2153518Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2154311Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2155058Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2155896Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2156749Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2157600Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2158272Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2158975Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2159817Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2160579Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2161451Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2162186Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2162957Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2163715Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2164563Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2165257Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2165708Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2166531Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2167352Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2168185Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2169052Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2169942Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2171016Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2172164Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2172589Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2173044Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2173920Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2174812Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2175588Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2176347Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2177102Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2177925Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2178647Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2179396Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2180157Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2180858Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2181554Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2182233Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2183127Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2183856Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2184559Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2185302Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2185999Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2186694Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2187368Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2188145Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2188928Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2189654Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2190054Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2190474Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2190869Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2191387Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2192172Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2192877Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2193652Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2194418Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2195287Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2196159Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2196924Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2197668Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2198397Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2199161Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2199886Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2200583Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2201273Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2202147Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2202886Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2203641Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2204475Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2205286Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2206155Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2207004Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2207892Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2208728Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2209494Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2210405Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2211243Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2212228Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2212682Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2213146Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2213696Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2214148Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2214579Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2214975Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2215358Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2215709Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2216076Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2216462Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2216842Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2217231Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2217619Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2218006Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2218416Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2218817Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2219215Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2219603Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2220037Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2220456Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2220854Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2221271Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2221686Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2222106Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2222532Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2222945Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2223333Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2223690Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2224169Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2224561Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2224936Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2225330Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2225786Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2226300Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2226742Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2227156Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2227625Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2228050Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2228507Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2228978Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2229420Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2229828Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2230270Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2230678Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2231090Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2231494Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2231913Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2232311Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2232747Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2233194Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2233610Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2234009Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2234477Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2234983Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2235537Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2236159Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2236768Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2237383Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2237973Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2238578Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2239180Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2239781Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2240331Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2240871Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2241317Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2241704Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2242088Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2242498Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2242902Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2243311Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2243755Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2244218Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2244696Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2245096Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2245490Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2245902Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2246321Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2246753Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2247217Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2247642Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2248054Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2248489Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2248913Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2249310Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2249718Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2250141Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2250531Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2250953Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2251416Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2252426Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2252813Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2253196Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2253595Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2254010Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2254429Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2254811Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2255175Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2255529Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2255909Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2256286Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2256645Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2257044Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2257427Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2257775Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2258132Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2258620Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2259002Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2259398Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2259771Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2260118Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2260480Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2260889Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2261308Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2261749Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2262205Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2262618Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2263027Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2263466Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2263897Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2264323Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2264725Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2265143Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2265620Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2266029Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2266424Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2266826Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2267295Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2267777Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2268249Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2268697Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2269143Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2269606Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2270038Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2270460Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2270877Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2271316Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2271769Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2272233Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2272749Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2273233Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2273658Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2274041Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2274445Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2274851Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2275224Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2275605Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2276057Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2276429Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2276798Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2277165Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2277542Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2277905Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2278308Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2278720Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2279080Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2279452Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2279840Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2280210Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2280581Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2280987Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2281369Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2281725Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2282113Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2282510Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2282937Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2283341Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2283715Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2284078Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2284506Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2284960Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2285390Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2285770Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2286176Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2286607Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2287003Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2287404Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2287817Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2288237Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2288654Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2289059Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2289472Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2289854Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2290237Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2290658Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2291095Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2292134Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2292903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2293359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2293926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2294521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2294753Z 
2022-11-23T01:44:15.2294865Z Running tests...
2022-11-23T01:44:15.2295259Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2295797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2296386Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2296953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2062
2022-11-23T01:44:15.2297383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2063
2022-11-23T01:44:15.2297984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2298432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2298995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2299469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2300047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2300493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2301120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2301605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2302059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2302559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2303214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2303904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2304428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2304886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2305420Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:44:15.2306252Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:44:15.2306918Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:44:15.2307729Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:44:15.2308391Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:44:15.2309209Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:44:15.2309866Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:44:15.2310686Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:44:15.2311330Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:44:15.2312210Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:44:15.2312867Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:44:15.2313682Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:44:15.2314326Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:44:15.2315140Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:44:15.2315799Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:44:15.2316611Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:44:15.2317079Z ok (5.724s)
2022-11-23T01:44:15.2317230Z 
2022-11-23T01:44:15.2317501Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2317829Z Ran 1 test in 5.724s
2022-11-23T01:44:15.2317990Z 
2022-11-23T01:44:15.2318066Z OK
2022-11-23T01:44:15.2318203Z 
2022-11-23T01:44:15.2318327Z Generating XML reports...
2022-11-23T01:44:15.2318996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012017.xml
2022-11-23T01:44:15.2319733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2320170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2320748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2321223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2321457Z 
2022-11-23T01:44:15.2321565Z Running tests...
2022-11-23T01:44:15.2321954Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2322488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2323026Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-11-23T01:44:15.2323339Z 
2022-11-23T01:44:15.2323583Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2323910Z Ran 1 test in 0.004s
2022-11-23T01:44:15.2324071Z 
2022-11-23T01:44:15.2324181Z OK (skipped=1)
2022-11-23T01:44:15.2324336Z 
2022-11-23T01:44:15.2324459Z Generating XML reports...
2022-11-23T01:44:15.2325046Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012026.xml
2022-11-23T01:44:15.2325762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2326211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2326770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2327237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2327471Z 
2022-11-23T01:44:15.2327635Z Running tests...
2022-11-23T01:44:15.2328046Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2328561Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2329073Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2329629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2210
2022-11-23T01:44:15.2330059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2211
2022-11-23T01:44:15.2330661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2331111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2332234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2332706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2333294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2333740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2334293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2334764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2335218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2335715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2336354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2337127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2337667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2338139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2338467Z ok (4.277s)
2022-11-23T01:44:15.2338620Z 
2022-11-23T01:44:15.2338893Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2339222Z Ran 1 test in 4.277s
2022-11-23T01:44:15.2339385Z 
2022-11-23T01:44:15.2339461Z OK
2022-11-23T01:44:15.2339596Z 
2022-11-23T01:44:15.2339721Z Generating XML reports...
2022-11-23T01:44:15.2340328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012028.xml
2022-11-23T01:44:15.2341037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2341473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2342048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2342518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2342748Z 
2022-11-23T01:44:15.2342844Z Running tests...
2022-11-23T01:44:15.2343247Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2343778Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2344305Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2345350Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2022-11-23T01:44:15.2345898Z 
2022-11-23T01:44:15.2346147Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2346477Z Ran 1 test in 1.638s
2022-11-23T01:44:15.2346637Z 
2022-11-23T01:44:15.2346748Z OK (skipped=1)
2022-11-23T01:44:15.2346902Z 
2022-11-23T01:44:15.2347094Z Generating XML reports...
2022-11-23T01:44:15.2347709Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012035.xml
2022-11-23T01:44:15.2348422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2348873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2349428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2349903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2350133Z 
2022-11-23T01:44:15.2350242Z Running tests...
2022-11-23T01:44:15.2350644Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2351159Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2351709Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T01:44:15.2352040Z 
2022-11-23T01:44:15.2352300Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2352611Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2352770Z 
2022-11-23T01:44:15.2352879Z OK (skipped=1)
2022-11-23T01:44:15.2353035Z 
2022-11-23T01:44:15.2353160Z Generating XML reports...
2022-11-23T01:44:15.2353761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012039.xml
2022-11-23T01:44:15.2354511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2354973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2355549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2356005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2356235Z 
2022-11-23T01:44:15.2356344Z Running tests...
2022-11-23T01:44:15.2356745Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2357271Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2357819Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T01:44:15.2358159Z 
2022-11-23T01:44:15.2358424Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2358748Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2358909Z 
2022-11-23T01:44:15.2358999Z OK (skipped=1)
2022-11-23T01:44:15.2359154Z 
2022-11-23T01:44:15.2359278Z Generating XML reports...
2022-11-23T01:44:15.2359881Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012041.xml
2022-11-23T01:44:15.2360596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2361030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2361602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2362074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2362303Z 
2022-11-23T01:44:15.2362410Z Running tests...
2022-11-23T01:44:15.2362797Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2363330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2363884Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2364399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2415
2022-11-23T01:44:15.2364916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2416
2022-11-23T01:44:15.2365522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2365971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2366526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2366996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2367577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2368006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2368575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2369045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2369502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2369983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2370637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2371369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2372449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2372911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2373410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr40rpkq
2022-11-23T01:44:15.2373958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr40rpkq/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2374473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkupd558d
2022-11-23T01:44:15.2375010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkupd558d/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2375933Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2376486Z   warnings.warn(
2022-11-23T01:44:15.2377224Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2377764Z   warnings.warn(
2022-11-23T01:44:15.2378145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2378631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2379096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2379575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2380048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2380512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2380994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2381472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2381943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2382505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2382978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2383445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2383793Z ok (6.537s)
2022-11-23T01:44:15.2383927Z 
2022-11-23T01:44:15.2384203Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2384531Z Ran 1 test in 6.538s
2022-11-23T01:44:15.2384693Z 
2022-11-23T01:44:15.2384792Z OK
2022-11-23T01:44:15.2384908Z 
2022-11-23T01:44:15.2385033Z Generating XML reports...
2022-11-23T01:44:15.2385640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012044.xml
2022-11-23T01:44:15.2386355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2386808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2387366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2387836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2388065Z 
2022-11-23T01:44:15.2388175Z Running tests...
2022-11-23T01:44:15.2388559Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2389157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2389736Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2390276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2534
2022-11-23T01:44:15.2390704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2535
2022-11-23T01:44:15.2391313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2391767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2392324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2392798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2393376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2393825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2394381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2394848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2395306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2395790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2396450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2397142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2397663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2398122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2398624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9n503asc
2022-11-23T01:44:15.2399165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9n503asc/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2399774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_cq1npu
2022-11-23T01:44:15.2400291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_cq1npu/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2401200Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2401748Z   warnings.warn(
2022-11-23T01:44:15.2402502Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2403027Z   warnings.warn(
2022-11-23T01:44:15.2403405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2403894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2404363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2404849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2405198Z ok (5.716s)
2022-11-23T01:44:15.2405346Z 
2022-11-23T01:44:15.2405618Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2405931Z Ran 1 test in 5.716s
2022-11-23T01:44:15.2406093Z 
2022-11-23T01:44:15.2406188Z OK
2022-11-23T01:44:15.2406321Z 
2022-11-23T01:44:15.2406504Z Generating XML reports...
2022-11-23T01:44:15.2407109Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012053.xml
2022-11-23T01:44:15.2407826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2408279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2408857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2409313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2409543Z 
2022-11-23T01:44:15.2409652Z Running tests...
2022-11-23T01:44:15.2410055Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2410565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2411136Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2412223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2653
2022-11-23T01:44:15.2412673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2654
2022-11-23T01:44:15.2413272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2413730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2414306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2414774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2415332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2415779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2416348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2416795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2417248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2417865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2418523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2419198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2419720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2420195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2420698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykmfgby8
2022-11-23T01:44:15.2421224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykmfgby8/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2421756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6v8em_l
2022-11-23T01:44:15.2422295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6v8em_l/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2423183Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2423736Z   warnings.warn(
2022-11-23T01:44:15.2424560Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2425114Z   warnings.warn(
2022-11-23T01:44:15.2425474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2425963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2426447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2426927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2427382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2427904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2428376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2428835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2429188Z ok (5.647s)
2022-11-23T01:44:15.2429336Z 
2022-11-23T01:44:15.2429609Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2429938Z Ran 1 test in 5.647s
2022-11-23T01:44:15.2430081Z 
2022-11-23T01:44:15.2430175Z OK
2022-11-23T01:44:15.2430314Z 
2022-11-23T01:44:15.2430440Z Generating XML reports...
2022-11-23T01:44:15.2431047Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012101.xml
2022-11-23T01:44:15.2431741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2432192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2432765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2433241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2433454Z 
2022-11-23T01:44:15.2433564Z Running tests...
2022-11-23T01:44:15.2433971Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2434502Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2435145Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2435709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2772
2022-11-23T01:44:15.2436155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2773
2022-11-23T01:44:15.2436761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2437197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2437825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2438298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2438879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2439308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2439873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2440335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2440770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2441267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2441977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2442678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2443186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2443664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2444164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxa5y150b
2022-11-23T01:44:15.2444706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxa5y150b/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2445220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn250st_m
2022-11-23T01:44:15.2445754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn250st_m/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2446661Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2447210Z   warnings.warn(
2022-11-23T01:44:15.2447948Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2448490Z   warnings.warn(
2022-11-23T01:44:15.2448865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2449337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2449690Z ok (5.925s)
2022-11-23T01:44:15.2449837Z 
2022-11-23T01:44:15.2450105Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2450434Z Ran 1 test in 5.925s
2022-11-23T01:44:15.2450579Z 
2022-11-23T01:44:15.2450672Z OK
2022-11-23T01:44:15.2450807Z 
2022-11-23T01:44:15.2450931Z Generating XML reports...
2022-11-23T01:44:15.2452217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012109.xml
2022-11-23T01:44:15.2452950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2453547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2454127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2454597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2454811Z 
2022-11-23T01:44:15.2454920Z Running tests...
2022-11-23T01:44:15.2455319Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2455850Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2456418Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2456978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2891
2022-11-23T01:44:15.2457430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2892
2022-11-23T01:44:15.2458034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2458467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2459041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2459510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2460143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2460602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2461177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2461647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2462086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2462580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2463241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2463926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2464433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2464906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2465407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnbp55o7
2022-11-23T01:44:15.2465934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnbp55o7/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2466468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvpgmxsv
2022-11-23T01:44:15.2467002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvpgmxsv/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2467908Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2468445Z   warnings.warn(
2022-11-23T01:44:15.2469196Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2469735Z   warnings.warn(
2022-11-23T01:44:15.2470112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2470650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2471130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2471612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2471963Z ok (6.262s)
2022-11-23T01:44:15.2472093Z 
2022-11-23T01:44:15.2472367Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2472699Z Ran 1 test in 6.262s
2022-11-23T01:44:15.2472858Z 
2022-11-23T01:44:15.2472952Z OK
2022-11-23T01:44:15.2473068Z 
2022-11-23T01:44:15.2473193Z Generating XML reports...
2022-11-23T01:44:15.2473802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012118.xml
2022-11-23T01:44:15.2474520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2474975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2475533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2476001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2476229Z 
2022-11-23T01:44:15.2476338Z Running tests...
2022-11-23T01:44:15.2476723Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2477305Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2477883Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2478419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3010
2022-11-23T01:44:15.2478848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3011
2022-11-23T01:44:15.2479458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2479907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2480465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2480935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2481519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2481962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2482519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2482984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2483439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2483919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2484578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2485269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2485792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2486252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2486751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz1j49at
2022-11-23T01:44:15.2487287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz1j49at/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2487880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcil3pe76
2022-11-23T01:44:15.2488398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcil3pe76/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2489303Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2489841Z   warnings.warn(
2022-11-23T01:44:15.2490592Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2491117Z   warnings.warn(
2022-11-23T01:44:15.2491671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2492177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2492637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2493115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2493462Z ok (6.146s)
2022-11-23T01:44:15.2493611Z 
2022-11-23T01:44:15.2493885Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2494196Z Ran 1 test in 6.146s
2022-11-23T01:44:15.2494357Z 
2022-11-23T01:44:15.2494531Z OK
2022-11-23T01:44:15.2494677Z 
2022-11-23T01:44:15.2494801Z Generating XML reports...
2022-11-23T01:44:15.2495396Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012127.xml
2022-11-23T01:44:15.2496107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2496566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2497136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2497588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2497817Z 
2022-11-23T01:44:15.2497926Z Running tests...
2022-11-23T01:44:15.2498325Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2498841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2499432Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2499994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3129
2022-11-23T01:44:15.2500441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3130
2022-11-23T01:44:15.2501034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2501485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2502055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2502524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2503090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2503535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2504104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2504550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2505093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2505590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2506254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2506922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2507450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2507921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2508419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ymdavx6
2022-11-23T01:44:15.2508943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ymdavx6/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2509481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvg9uzun
2022-11-23T01:44:15.2510017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvg9uzun/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2510909Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2511457Z   warnings.warn(
2022-11-23T01:44:15.2512264Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.2512812Z   warnings.warn(
2022-11-23T01:44:15.2513172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2513661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2514145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2514625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2514954Z ok (5.597s)
2022-11-23T01:44:15.2515103Z 
2022-11-23T01:44:15.2515375Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2515700Z Ran 1 test in 5.597s
2022-11-23T01:44:15.2515863Z 
2022-11-23T01:44:15.2515939Z OK
2022-11-23T01:44:15.2516077Z 
2022-11-23T01:44:15.2516203Z Generating XML reports...
2022-11-23T01:44:15.2516811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012135.xml
2022-11-23T01:44:15.2517520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2517959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2518531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2519001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2519231Z 
2022-11-23T01:44:15.2519322Z Running tests...
2022-11-23T01:44:15.2519724Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2520254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2520813Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2521863Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2022-11-23T01:44:15.2522452Z 
2022-11-23T01:44:15.2522720Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2523047Z Ran 1 test in 1.641s
2022-11-23T01:44:15.2523209Z 
2022-11-23T01:44:15.2523316Z OK (skipped=1)
2022-11-23T01:44:15.2523471Z 
2022-11-23T01:44:15.2523578Z Generating XML reports...
2022-11-23T01:44:15.2524182Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012143.xml
2022-11-23T01:44:15.2524896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2525349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2525908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2526378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2526612Z 
2022-11-23T01:44:15.2526720Z Running tests...
2022-11-23T01:44:15.2527103Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2527680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2528228Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2528757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3282
2022-11-23T01:44:15.2529246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3283
2022-11-23T01:44:15.2529860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2530314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2530871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2531346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2532130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2532578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2533132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2533605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2534062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2534557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2535198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2535887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2536411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2536868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2537213Z ok (4.321s)
2022-11-23T01:44:15.2537362Z 
2022-11-23T01:44:15.2537628Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2537958Z Ran 1 test in 4.321s
2022-11-23T01:44:15.2538101Z 
2022-11-23T01:44:15.2538194Z OK
2022-11-23T01:44:15.2538327Z 
2022-11-23T01:44:15.2538451Z Generating XML reports...
2022-11-23T01:44:15.2539058Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012147.xml
2022-11-23T01:44:15.2539757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2540315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2540897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2541365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2541577Z 
2022-11-23T01:44:15.2541685Z Running tests...
2022-11-23T01:44:15.2542084Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2542618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2543162Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2544276Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2022-11-23T01:44:15.2544806Z 
2022-11-23T01:44:15.2545069Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2545398Z Ran 1 test in 1.637s
2022-11-23T01:44:15.2545559Z 
2022-11-23T01:44:15.2545667Z OK (skipped=1)
2022-11-23T01:44:15.2545804Z 
2022-11-23T01:44:15.2545931Z Generating XML reports...
2022-11-23T01:44:15.2546602Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012154.xml
2022-11-23T01:44:15.2547337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2547772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2548339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2548812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2549042Z 
2022-11-23T01:44:15.2549150Z Running tests...
2022-11-23T01:44:15.2549536Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2550063Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2550589Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2551080Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3421
2022-11-23T01:44:15.2551523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3422
2022-11-23T01:44:15.2552123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2552577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2553138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2553606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2554182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2554609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2555181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2555650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2556107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2556589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2557316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2558006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2558531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2558987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2559328Z ok (5.772s)
2022-11-23T01:44:15.2559477Z 
2022-11-23T01:44:15.2559748Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2560059Z Ran 1 test in 5.772s
2022-11-23T01:44:15.2560219Z 
2022-11-23T01:44:15.2560312Z OK
2022-11-23T01:44:15.2560448Z 
2022-11-23T01:44:15.2560572Z Generating XML reports...
2022-11-23T01:44:15.2561175Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012159.xml
2022-11-23T01:44:15.2561875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2562325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2562895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2563350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2563581Z 
2022-11-23T01:44:15.2563688Z Running tests...
2022-11-23T01:44:15.2564150Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2564700Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2565191Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T01:44:15.2565480Z 
2022-11-23T01:44:15.2565744Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2566070Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2566231Z 
2022-11-23T01:44:15.2566321Z OK (skipped=1)
2022-11-23T01:44:15.2566477Z 
2022-11-23T01:44:15.2566600Z Generating XML reports...
2022-11-23T01:44:15.2567198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012207.xml
2022-11-23T01:44:15.2567908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2568347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2568921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2569393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2569625Z 
2022-11-23T01:44:15.2569732Z Running tests...
2022-11-23T01:44:15.2570120Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2570648Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2571107Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2571752Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2572247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3569
2022-11-23T01:44:15.2572703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3570
2022-11-23T01:44:15.2573322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2573756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2574329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2574890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2575453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2575897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2576475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2576938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2577380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2577881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2578535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2579232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2579737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2580212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2580711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33it033t
2022-11-23T01:44:15.2581234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33it033t/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2581834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuh4hhwe4
2022-11-23T01:44:15.2582387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuh4hhwe4/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2582766Z ok (6.006s)
2022-11-23T01:44:15.2582898Z 
2022-11-23T01:44:15.2583171Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2583508Z Ran 1 test in 6.006s
2022-11-23T01:44:15.2583668Z 
2022-11-23T01:44:15.2583762Z OK
2022-11-23T01:44:15.2583895Z 
2022-11-23T01:44:15.2584001Z Generating XML reports...
2022-11-23T01:44:15.2584607Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012209.xml
2022-11-23T01:44:15.2585321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2585771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2586328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2586794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2587024Z 
2022-11-23T01:44:15.2587132Z Running tests...
2022-11-23T01:44:15.2587513Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2588051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2588530Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2589027Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2589495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3688
2022-11-23T01:44:15.2589941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3689
2022-11-23T01:44:15.2590550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2590984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2591559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2592095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2592677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2593103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2593674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2594142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2594598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2595076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2595732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2596421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2596924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2597396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2597894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1mmeqnt
2022-11-23T01:44:15.2598435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1mmeqnt/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2598998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb6yzgb_
2022-11-23T01:44:15.2599547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb6yzgb_/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2600062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2600547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2600885Z ok (6.050s)
2022-11-23T01:44:15.2601033Z 
2022-11-23T01:44:15.2601305Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2601633Z Ran 1 test in 6.050s
2022-11-23T01:44:15.2601794Z 
2022-11-23T01:44:15.2601871Z OK
2022-11-23T01:44:15.2602006Z 
2022-11-23T01:44:15.2602131Z Generating XML reports...
2022-11-23T01:44:15.2602733Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012218.xml
2022-11-23T01:44:15.2603432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2603887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2604461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2604930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2605162Z 
2022-11-23T01:44:15.2605253Z Running tests...
2022-11-23T01:44:15.2605656Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2606185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2606679Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2607197Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2607702Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3807
2022-11-23T01:44:15.2608146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3808
2022-11-23T01:44:15.2608738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2609188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2609825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2610292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2610851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2611297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2612064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2612517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2612972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2613471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2614127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2614800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2615320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2615792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2616367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3qt9ll8
2022-11-23T01:44:15.2616910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3qt9ll8/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2617439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdnl7lnb
2022-11-23T01:44:15.2617978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdnl7lnb/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2618481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2618970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.2619322Z ok (6.149s)
2022-11-23T01:44:15.2619469Z 
2022-11-23T01:44:15.2619745Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2620057Z Ran 1 test in 6.149s
2022-11-23T01:44:15.2620219Z 
2022-11-23T01:44:15.2620315Z OK
2022-11-23T01:44:15.2620450Z 
2022-11-23T01:44:15.2620578Z Generating XML reports...
2022-11-23T01:44:15.2621168Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012226.xml
2022-11-23T01:44:15.2621880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2622336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2622913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2623367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2623595Z 
2022-11-23T01:44:15.2623705Z Running tests...
2022-11-23T01:44:15.2624104Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2624620Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2625101Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:44:15.2625602Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2626083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3926
2022-11-23T01:44:15.2626511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3927
2022-11-23T01:44:15.2627197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2627692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2628255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2628728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2629309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2629759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2630317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2630784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2631239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2631739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2632382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2633076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2633603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2634107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2634624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfv2_n6_
2022-11-23T01:44:15.2635159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfv2_n6_/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2635692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90vyqnru
2022-11-23T01:44:15.2636206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90vyqnru/_remote_module_non_scriptable.py
2022-11-23T01:44:15.2636591Z ok (6.148s)
2022-11-23T01:44:15.2636740Z 
2022-11-23T01:44:15.2637011Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2637335Z Ran 1 test in 6.148s
2022-11-23T01:44:15.2637479Z 
2022-11-23T01:44:15.2637573Z OK
2022-11-23T01:44:15.2637710Z 
2022-11-23T01:44:15.2637837Z Generating XML reports...
2022-11-23T01:44:15.2638452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012235.xml
2022-11-23T01:44:15.2639155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2639606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2640183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2640648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2640859Z 
2022-11-23T01:44:15.2640968Z Running tests...
2022-11-23T01:44:15.2641368Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2641897Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2642389Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2642674Z 
2022-11-23T01:44:15.2642937Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2643262Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2643421Z 
2022-11-23T01:44:15.2643529Z OK (skipped=1)
2022-11-23T01:44:15.2643665Z 
2022-11-23T01:44:15.2643789Z Generating XML reports...
2022-11-23T01:44:15.2644458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012244.xml
2022-11-23T01:44:15.2645169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2645598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2646171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2646639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2646872Z 
2022-11-23T01:44:15.2646983Z Running tests...
2022-11-23T01:44:15.2647370Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2647899Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2648443Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T01:44:15.2648766Z 
2022-11-23T01:44:15.2649028Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2649339Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2649500Z 
2022-11-23T01:44:15.2649607Z OK (skipped=1)
2022-11-23T01:44:15.2649763Z 
2022-11-23T01:44:15.2649886Z Generating XML reports...
2022-11-23T01:44:15.2650470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012246.xml
2022-11-23T01:44:15.2651228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2651875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2652462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2652918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2653154Z 
2022-11-23T01:44:15.2653263Z Running tests...
2022-11-23T01:44:15.2653664Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2654176Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2654722Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T01:44:15.2655042Z 
2022-11-23T01:44:15.2655304Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2655633Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2655777Z 
2022-11-23T01:44:15.2655885Z OK (skipped=1)
2022-11-23T01:44:15.2656038Z 
2022-11-23T01:44:15.2656163Z Generating XML reports...
2022-11-23T01:44:15.2656766Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012249.xml
2022-11-23T01:44:15.2657455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2657906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2658482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2658945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2659171Z 
2022-11-23T01:44:15.2659263Z Running tests...
2022-11-23T01:44:15.2659664Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2660197Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2660718Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T01:44:15.2661037Z 
2022-11-23T01:44:15.2661300Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2661729Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2661889Z 
2022-11-23T01:44:15.2661998Z OK (skipped=1)
2022-11-23T01:44:15.2662149Z 
2022-11-23T01:44:15.2662256Z Generating XML reports...
2022-11-23T01:44:15.2662864Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012251.xml
2022-11-23T01:44:15.2663575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2664015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2664599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2665072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2665301Z 
2022-11-23T01:44:15.2665410Z Running tests...
2022-11-23T01:44:15.2665793Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2666322Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2666858Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T01:44:15.2667175Z 
2022-11-23T01:44:15.2667420Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2667747Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2667906Z 
2022-11-23T01:44:15.2668013Z OK (skipped=1)
2022-11-23T01:44:15.2668032Z 
2022-11-23T01:44:15.2668156Z Generating XML reports...
2022-11-23T01:44:15.2668673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012253.xml
2022-11-23T01:44:15.2669043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2669223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2669609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2669802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2669822Z 
2022-11-23T01:44:15.2669931Z Running tests...
2022-11-23T01:44:15.2670190Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2670504Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2670801Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-11-23T01:44:15.2670821Z 
2022-11-23T01:44:15.2671081Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2671177Z Ran 1 test in 0.003s
2022-11-23T01:44:15.2671196Z 
2022-11-23T01:44:15.2671302Z OK (skipped=1)
2022-11-23T01:44:15.2671321Z 
2022-11-23T01:44:15.2671443Z Generating XML reports...
2022-11-23T01:44:15.2671895Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012256.xml
2022-11-23T01:44:15.2672262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2672438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2672815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2673009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2673028Z 
2022-11-23T01:44:15.2673136Z Running tests...
2022-11-23T01:44:15.2673382Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2673695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2673963Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2674035Z 
2022-11-23T01:44:15.2674310Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2674423Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2674443Z 
2022-11-23T01:44:15.2674551Z OK (skipped=1)
2022-11-23T01:44:15.2674570Z 
2022-11-23T01:44:15.2674693Z Generating XML reports...
2022-11-23T01:44:15.2675141Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012258.xml
2022-11-23T01:44:15.2675499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2675676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2676054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2676244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2676266Z 
2022-11-23T01:44:15.2676373Z Running tests...
2022-11-23T01:44:15.2676630Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2676944Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2677199Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2677416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4276
2022-11-23T01:44:15.2677660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4277
2022-11-23T01:44:15.2678051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2678227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2678604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2678798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2679165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2679339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2679715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2679892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2680140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2680377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2680776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2681176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2681404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2681632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2681965Z STAGE:2022-11-23 01:23:05 4276:4276 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2682286Z STAGE:2022-11-23 01:23:05 4277:4277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2682601Z STAGE:2022-11-23 01:23:06 4276:4276 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2682936Z STAGE:2022-11-23 01:23:06 4277:4277 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2683280Z STAGE:2022-11-23 01:23:06 4276:4276 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2683685Z STAGE:2022-11-23 01:23:06 4277:4277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2684010Z STAGE:2022-11-23 01:23:06 4276:4276 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2684333Z STAGE:2022-11-23 01:23:06 4276:4276 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2684670Z STAGE:2022-11-23 01:23:06 4276:4276 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2684992Z STAGE:2022-11-23 01:23:06 4277:4277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2685320Z STAGE:2022-11-23 01:23:06 4277:4277 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2685647Z STAGE:2022-11-23 01:23:06 4277:4277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2685750Z ok (6.484s)
2022-11-23T01:44:15.2685770Z 
2022-11-23T01:44:15.2686035Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2686146Z Ran 1 test in 6.484s
2022-11-23T01:44:15.2686166Z 
2022-11-23T01:44:15.2686258Z OK
2022-11-23T01:44:15.2686277Z 
2022-11-23T01:44:15.2686398Z Generating XML reports...
2022-11-23T01:44:15.2686849Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012301.xml
2022-11-23T01:44:15.2687220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2687442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2687815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2688012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2688032Z 
2022-11-23T01:44:15.2688141Z Running tests...
2022-11-23T01:44:15.2688407Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2688720Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2688987Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2689205Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4395
2022-11-23T01:44:15.2689416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4396
2022-11-23T01:44:15.2689773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2689949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2690326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2690516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2690884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2691057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2691593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2691796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2692050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2692272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2692678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2693076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2693391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2693618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2693954Z STAGE:2022-11-23 01:23:14 4395:4395 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2694276Z STAGE:2022-11-23 01:23:14 4396:4396 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2694609Z STAGE:2022-11-23 01:23:15 4395:4395 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2694925Z STAGE:2022-11-23 01:23:15 4396:4396 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2695271Z STAGE:2022-11-23 01:23:15 4395:4395 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2695610Z STAGE:2022-11-23 01:23:15 4396:4396 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2695937Z STAGE:2022-11-23 01:23:15 4395:4395 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2696266Z STAGE:2022-11-23 01:23:15 4395:4395 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2696607Z STAGE:2022-11-23 01:23:15 4395:4395 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2696929Z STAGE:2022-11-23 01:23:15 4396:4396 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2697356Z STAGE:2022-11-23 01:23:15 4396:4396 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2697714Z STAGE:2022-11-23 01:23:15 4396:4396 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2697800Z ok (6.503s)
2022-11-23T01:44:15.2697837Z 
2022-11-23T01:44:15.2698084Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2698202Z Ran 1 test in 6.503s
2022-11-23T01:44:15.2698221Z 
2022-11-23T01:44:15.2698315Z OK
2022-11-23T01:44:15.2698335Z 
2022-11-23T01:44:15.2698461Z Generating XML reports...
2022-11-23T01:44:15.2698911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012310.xml
2022-11-23T01:44:15.2699282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2699459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2699844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2700020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2700040Z 
2022-11-23T01:44:15.2700147Z Running tests...
2022-11-23T01:44:15.2700411Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2700725Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2700993Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2701013Z 
2022-11-23T01:44:15.2701274Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2701383Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2701403Z 
2022-11-23T01:44:15.2701511Z OK (skipped=1)
2022-11-23T01:44:15.2701530Z 
2022-11-23T01:44:15.2701636Z Generating XML reports...
2022-11-23T01:44:15.2702088Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012319.xml
2022-11-23T01:44:15.2702460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2702634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2703013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2703266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2703286Z 
2022-11-23T01:44:15.2703395Z Running tests...
2022-11-23T01:44:15.2703656Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2703970Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2704217Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2704258Z 
2022-11-23T01:44:15.2704502Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2704616Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2704635Z 
2022-11-23T01:44:15.2704741Z OK (skipped=1)
2022-11-23T01:44:15.2704760Z 
2022-11-23T01:44:15.2704883Z Generating XML reports...
2022-11-23T01:44:15.2705330Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012321.xml
2022-11-23T01:44:15.2705702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2705878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2706258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2706433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2706452Z 
2022-11-23T01:44:15.2706604Z Running tests...
2022-11-23T01:44:15.2706880Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2707194Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2707471Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2707694Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4580
2022-11-23T01:44:15.2707912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4581
2022-11-23T01:44:15.2708285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2708445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2708819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2709013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2709381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2709555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2709929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2710122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2710366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2710602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2710985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2711385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2711614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2711841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2712173Z STAGE:2022-11-23 01:23:28 4581:4581 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2712556Z STAGE:2022-11-23 01:23:28 4580:4580 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2712886Z STAGE:2022-11-23 01:23:29 4580:4580 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2713219Z STAGE:2022-11-23 01:23:29 4581:4581 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2713565Z STAGE:2022-11-23 01:23:29 4580:4580 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2713893Z STAGE:2022-11-23 01:23:29 4581:4581 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2713998Z ok (6.320s)
2022-11-23T01:44:15.2714018Z 
2022-11-23T01:44:15.2714284Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2714395Z Ran 1 test in 6.321s
2022-11-23T01:44:15.2714415Z 
2022-11-23T01:44:15.2714508Z OK
2022-11-23T01:44:15.2714531Z 
2022-11-23T01:44:15.2714655Z Generating XML reports...
2022-11-23T01:44:15.2715103Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012323.xml
2022-11-23T01:44:15.2715471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2715629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2716007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2716247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2716268Z 
2022-11-23T01:44:15.2716383Z Running tests...
2022-11-23T01:44:15.2716655Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2716968Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2717252Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2717472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4699
2022-11-23T01:44:15.2717682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4700
2022-11-23T01:44:15.2718036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2718212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2718591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2718782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2719151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2719328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2719704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2719896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2720127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2720366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2720772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2721169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2721398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2721682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2722017Z STAGE:2022-11-23 01:23:37 4699:4699 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2722336Z STAGE:2022-11-23 01:23:37 4700:4700 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2722664Z STAGE:2022-11-23 01:23:38 4699:4699 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2722978Z STAGE:2022-11-23 01:23:38 4700:4700 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2723327Z STAGE:2022-11-23 01:23:38 4699:4699 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2723666Z STAGE:2022-11-23 01:23:38 4700:4700 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2723768Z ok (6.514s)
2022-11-23T01:44:15.2723788Z 
2022-11-23T01:44:15.2724052Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2724165Z Ran 1 test in 6.515s
2022-11-23T01:44:15.2724184Z 
2022-11-23T01:44:15.2724276Z OK
2022-11-23T01:44:15.2724296Z 
2022-11-23T01:44:15.2724420Z Generating XML reports...
2022-11-23T01:44:15.2724871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012332.xml
2022-11-23T01:44:15.2725224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2725400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2725826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2726028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2726047Z 
2022-11-23T01:44:15.2726157Z Running tests...
2022-11-23T01:44:15.2726420Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2726737Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2727002Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2727801Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2022-11-23T01:44:15.2727823Z 
2022-11-23T01:44:15.2728072Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2728183Z Ran 1 test in 1.638s
2022-11-23T01:44:15.2728202Z 
2022-11-23T01:44:15.2728310Z OK (skipped=1)
2022-11-23T01:44:15.2728329Z 
2022-11-23T01:44:15.2728455Z Generating XML reports...
2022-11-23T01:44:15.2728903Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012341.xml
2022-11-23T01:44:15.2729275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2729453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2729836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2730026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2730045Z 
2022-11-23T01:44:15.2730139Z Running tests...
2022-11-23T01:44:15.2730400Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2730712Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2730984Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2731975Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.639s)
2022-11-23T01:44:15.2731998Z 
2022-11-23T01:44:15.2732271Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2732386Z Ran 1 test in 1.640s
2022-11-23T01:44:15.2732405Z 
2022-11-23T01:44:15.2732515Z OK (skipped=1)
2022-11-23T01:44:15.2732535Z 
2022-11-23T01:44:15.2732665Z Generating XML reports...
2022-11-23T01:44:15.2733117Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012346.xml
2022-11-23T01:44:15.2733473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2733653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2734037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2734232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2734252Z 
2022-11-23T01:44:15.2734362Z Running tests...
2022-11-23T01:44:15.2734625Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2734937Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2735288Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2735501Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4886
2022-11-23T01:44:15.2735716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4887
2022-11-23T01:44:15.2736092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2736272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2736651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2736842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2737210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2737384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2737821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2738000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2738251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2738495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2738894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2739290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2739519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2739751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2739855Z ok (5.754s)
2022-11-23T01:44:15.2739875Z 
2022-11-23T01:44:15.2740139Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2740236Z Ran 1 test in 5.754s
2022-11-23T01:44:15.2740255Z 
2022-11-23T01:44:15.2740348Z OK
2022-11-23T01:44:15.2740367Z 
2022-11-23T01:44:15.2740492Z Generating XML reports...
2022-11-23T01:44:15.2741027Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012350.xml
2022-11-23T01:44:15.2741399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2741576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2741954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2742148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2742167Z 
2022-11-23T01:44:15.2742260Z Running tests...
2022-11-23T01:44:15.2742522Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2742833Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2743104Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2743324Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5001
2022-11-23T01:44:15.2743533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5002
2022-11-23T01:44:15.2743901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2744075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2744482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2744684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2745054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2745229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2745609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2745802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2746051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2746292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2746692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2747069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2747299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2747530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2747774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.2748008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.2748401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.2748787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.2749031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:44:15.2749265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:44:15.2749639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.2750032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.2750326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:44:15.2750561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:44:15.2750953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T01:44:15.2751346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T01:44:15.2751450Z ok (6.552s)
2022-11-23T01:44:15.2751470Z 
2022-11-23T01:44:15.2751736Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2751850Z Ran 1 test in 6.552s
2022-11-23T01:44:15.2751869Z 
2022-11-23T01:44:15.2751947Z OK
2022-11-23T01:44:15.2751981Z 
2022-11-23T01:44:15.2752092Z Generating XML reports...
2022-11-23T01:44:15.2752545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012358.xml
2022-11-23T01:44:15.2752914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2753087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2753467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2753705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2753726Z 
2022-11-23T01:44:15.2753842Z Running tests...
2022-11-23T01:44:15.2754108Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2754406Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2754664Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2754887Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5152
2022-11-23T01:44:15.2755104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5153
2022-11-23T01:44:15.2755470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2755646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2756026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2756218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2756568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2756744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2757125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2757317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2757562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2757799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2758198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2758590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2758822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2759030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2759187Z ok (5.544s)
2022-11-23T01:44:15.2759206Z 
2022-11-23T01:44:15.2759474Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2759587Z Ran 1 test in 5.544s
2022-11-23T01:44:15.2759606Z 
2022-11-23T01:44:15.2759698Z OK
2022-11-23T01:44:15.2759717Z 
2022-11-23T01:44:15.2759840Z Generating XML reports...
2022-11-23T01:44:15.2760294Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012407.xml
2022-11-23T01:44:15.2760665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2760844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2761209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2761400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2761423Z 
2022-11-23T01:44:15.2761530Z Running tests...
2022-11-23T01:44:15.2761790Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2762104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2762519Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2762539Z 
2022-11-23T01:44:15.2762793Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2762963Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2762984Z 
2022-11-23T01:44:15.2763079Z OK (skipped=1)
2022-11-23T01:44:15.2763118Z 
2022-11-23T01:44:15.2763225Z Generating XML reports...
2022-11-23T01:44:15.2763672Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012415.xml
2022-11-23T01:44:15.2764038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2764218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2764594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2764784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2764803Z 
2022-11-23T01:44:15.2764911Z Running tests...
2022-11-23T01:44:15.2765167Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2765467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2765881Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2765900Z 
2022-11-23T01:44:15.2766157Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2766271Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2766291Z 
2022-11-23T01:44:15.2766400Z OK (skipped=1)
2022-11-23T01:44:15.2766419Z 
2022-11-23T01:44:15.2766542Z Generating XML reports...
2022-11-23T01:44:15.2766986Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012418.xml
2022-11-23T01:44:15.2767353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2767532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2767896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2768090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2768109Z 
2022-11-23T01:44:15.2768216Z Running tests...
2022-11-23T01:44:15.2768479Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2768850Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2769282Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2769302Z 
2022-11-23T01:44:15.2769564Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2769677Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2769696Z 
2022-11-23T01:44:15.2769803Z OK (skipped=1)
2022-11-23T01:44:15.2769822Z 
2022-11-23T01:44:15.2769931Z Generating XML reports...
2022-11-23T01:44:15.2770380Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012420.xml
2022-11-23T01:44:15.2770750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2770927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2771310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2771670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2771691Z 
2022-11-23T01:44:15.2771805Z Running tests...
2022-11-23T01:44:15.2772070Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2772366Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2772862Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2772885Z 
2022-11-23T01:44:15.2773152Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2773265Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2773285Z 
2022-11-23T01:44:15.2773391Z OK (skipped=1)
2022-11-23T01:44:15.2773410Z 
2022-11-23T01:44:15.2773539Z Generating XML reports...
2022-11-23T01:44:15.2773982Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012422.xml
2022-11-23T01:44:15.2774351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2774528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2774890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2775090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2775109Z 
2022-11-23T01:44:15.2775219Z Running tests...
2022-11-23T01:44:15.2775480Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2775792Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2776202Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2776225Z 
2022-11-23T01:44:15.2776484Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2776594Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2776613Z 
2022-11-23T01:44:15.2776721Z OK (skipped=1)
2022-11-23T01:44:15.2776740Z 
2022-11-23T01:44:15.2776846Z Generating XML reports...
2022-11-23T01:44:15.2777287Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012425.xml
2022-11-23T01:44:15.2777659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2777835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2778212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2778478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2778498Z 
2022-11-23T01:44:15.2778607Z Running tests...
2022-11-23T01:44:15.2778870Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2779180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2779571Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2779590Z 
2022-11-23T01:44:15.2779856Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2779968Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2779987Z 
2022-11-23T01:44:15.2780093Z OK (skipped=1)
2022-11-23T01:44:15.2780112Z 
2022-11-23T01:44:15.2780236Z Generating XML reports...
2022-11-23T01:44:15.2780684Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012427.xml
2022-11-23T01:44:15.2781056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2781231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2781612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2781787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2781807Z 
2022-11-23T01:44:15.2781915Z Running tests...
2022-11-23T01:44:15.2782221Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2782543Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2782962Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2782982Z 
2022-11-23T01:44:15.2783245Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2783356Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2783375Z 
2022-11-23T01:44:15.2783483Z OK (skipped=1)
2022-11-23T01:44:15.2783502Z 
2022-11-23T01:44:15.2783607Z Generating XML reports...
2022-11-23T01:44:15.2784052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012429.xml
2022-11-23T01:44:15.2784421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2784599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2784977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2785169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2785188Z 
2022-11-23T01:44:15.2785296Z Running tests...
2022-11-23T01:44:15.2785563Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2785874Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2786263Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2786300Z 
2022-11-23T01:44:15.2786542Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2786652Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2786672Z 
2022-11-23T01:44:15.2786776Z OK (skipped=1)
2022-11-23T01:44:15.2786798Z 
2022-11-23T01:44:15.2786922Z Generating XML reports...
2022-11-23T01:44:15.2787370Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012432.xml
2022-11-23T01:44:15.2787740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2787972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2788357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2788532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2788552Z 
2022-11-23T01:44:15.2788660Z Running tests...
2022-11-23T01:44:15.2788924Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2789237Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2789635Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2789655Z 
2022-11-23T01:44:15.2789915Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2790026Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2790045Z 
2022-11-23T01:44:15.2790155Z OK (skipped=1)
2022-11-23T01:44:15.2790175Z 
2022-11-23T01:44:15.2790297Z Generating XML reports...
2022-11-23T01:44:15.2790727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012434.xml
2022-11-23T01:44:15.2791095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2791269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2791693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2791896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2791915Z 
2022-11-23T01:44:15.2792024Z Running tests...
2022-11-23T01:44:15.2792286Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2792599Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2792892Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2792930Z 
2022-11-23T01:44:15.2793168Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2793277Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2793297Z 
2022-11-23T01:44:15.2793402Z OK (skipped=1)
2022-11-23T01:44:15.2793421Z 
2022-11-23T01:44:15.2793541Z Generating XML reports...
2022-11-23T01:44:15.2793989Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012437.xml
2022-11-23T01:44:15.2794359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2794534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2794909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2795086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2795121Z 
2022-11-23T01:44:15.2795211Z Running tests...
2022-11-23T01:44:15.2795467Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2795777Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2796170Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2796193Z 
2022-11-23T01:44:15.2796450Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2796563Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2796582Z 
2022-11-23T01:44:15.2796689Z OK (skipped=1)
2022-11-23T01:44:15.2796708Z 
2022-11-23T01:44:15.2796830Z Generating XML reports...
2022-11-23T01:44:15.2797260Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012439.xml
2022-11-23T01:44:15.2797700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2797878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2798258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2798451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2798470Z 
2022-11-23T01:44:15.2798582Z Running tests...
2022-11-23T01:44:15.2798844Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2799157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2799562Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2799586Z 
2022-11-23T01:44:15.2799829Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2799939Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2799958Z 
2022-11-23T01:44:15.2800064Z OK (skipped=1)
2022-11-23T01:44:15.2800083Z 
2022-11-23T01:44:15.2800205Z Generating XML reports...
2022-11-23T01:44:15.2800652Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012441.xml
2022-11-23T01:44:15.2801066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2801252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2801629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2801821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2801844Z 
2022-11-23T01:44:15.2801936Z Running tests...
2022-11-23T01:44:15.2802196Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2802508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2802904Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.2802924Z 
2022-11-23T01:44:15.2803180Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2803290Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2803309Z 
2022-11-23T01:44:15.2803419Z OK (skipped=1)
2022-11-23T01:44:15.2803440Z 
2022-11-23T01:44:15.2803563Z Generating XML reports...
2022-11-23T01:44:15.2803999Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012444.xml
2022-11-23T01:44:15.2804369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2804548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2804928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2805118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2805137Z 
2022-11-23T01:44:15.2805244Z Running tests...
2022-11-23T01:44:15.2805503Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2805817Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2806108Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2806128Z 
2022-11-23T01:44:15.2806369Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2806480Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2806548Z 
2022-11-23T01:44:15.2806663Z OK (skipped=1)
2022-11-23T01:44:15.2806682Z 
2022-11-23T01:44:15.2806805Z Generating XML reports...
2022-11-23T01:44:15.2807254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012446.xml
2022-11-23T01:44:15.2807623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2807800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2808187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2808378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2808399Z 
2022-11-23T01:44:15.2808490Z Running tests...
2022-11-23T01:44:15.2808748Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2809060Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2809338Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2809358Z 
2022-11-23T01:44:15.2809615Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2809725Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2809744Z 
2022-11-23T01:44:15.2809850Z OK (skipped=1)
2022-11-23T01:44:15.2809870Z 
2022-11-23T01:44:15.2809990Z Generating XML reports...
2022-11-23T01:44:15.2847572Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012449.xml
2022-11-23T01:44:15.2848078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2848262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2848638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2848842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2848865Z 
2022-11-23T01:44:15.2848978Z Running tests...
2022-11-23T01:44:15.2849253Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2849572Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2849852Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2849876Z 
2022-11-23T01:44:15.2850143Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2850259Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2850279Z 
2022-11-23T01:44:15.2850369Z OK (skipped=1)
2022-11-23T01:44:15.2850407Z 
2022-11-23T01:44:15.2850514Z Generating XML reports...
2022-11-23T01:44:15.2850973Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012451.xml
2022-11-23T01:44:15.2851351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2851826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2852228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2852424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2852445Z 
2022-11-23T01:44:15.2852559Z Running tests...
2022-11-23T01:44:15.2852826Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2853125Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2853409Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2853518Z 
2022-11-23T01:44:15.2853792Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2853905Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2853925Z 
2022-11-23T01:44:15.2854034Z OK (skipped=1)
2022-11-23T01:44:15.2854053Z 
2022-11-23T01:44:15.2854179Z Generating XML reports...
2022-11-23T01:44:15.2854631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012453.xml
2022-11-23T01:44:15.2855007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2855188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2855556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2855752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2855776Z 
2022-11-23T01:44:15.2855886Z Running tests...
2022-11-23T01:44:15.2856153Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2856469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2856746Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2856766Z 
2022-11-23T01:44:15.2857029Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2857144Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2857163Z 
2022-11-23T01:44:15.2857317Z OK (skipped=1)
2022-11-23T01:44:15.2857359Z 
2022-11-23T01:44:15.2857473Z Generating XML reports...
2022-11-23T01:44:15.2857929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012456.xml
2022-11-23T01:44:15.2858306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2858487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2858868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2859061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2859080Z 
2022-11-23T01:44:15.2859193Z Running tests...
2022-11-23T01:44:15.2859458Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2859760Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2860031Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2860050Z 
2022-11-23T01:44:15.2860311Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2860425Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2860448Z 
2022-11-23T01:44:15.2860557Z OK (skipped=1)
2022-11-23T01:44:15.2860576Z 
2022-11-23T01:44:15.2860701Z Generating XML reports...
2022-11-23T01:44:15.2861148Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012458.xml
2022-11-23T01:44:15.2861518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2861697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2862063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2862256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2862275Z 
2022-11-23T01:44:15.2862384Z Running tests...
2022-11-23T01:44:15.2862648Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2862959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2863301Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2863321Z 
2022-11-23T01:44:15.2863586Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2863700Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2863719Z 
2022-11-23T01:44:15.2863828Z OK (skipped=1)
2022-11-23T01:44:15.2863847Z 
2022-11-23T01:44:15.2863952Z Generating XML reports...
2022-11-23T01:44:15.2864401Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012500.xml
2022-11-23T01:44:15.2864772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2864949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2865326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2865523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2865542Z 
2022-11-23T01:44:15.2865653Z Running tests...
2022-11-23T01:44:15.2865918Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2866212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2866489Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2866561Z 
2022-11-23T01:44:15.2866834Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2866948Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2866967Z 
2022-11-23T01:44:15.2867076Z OK (skipped=1)
2022-11-23T01:44:15.2867095Z 
2022-11-23T01:44:15.2867220Z Generating XML reports...
2022-11-23T01:44:15.2867670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012503.xml
2022-11-23T01:44:15.2868047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2868228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2868590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2868786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2868806Z 
2022-11-23T01:44:15.2868920Z Running tests...
2022-11-23T01:44:15.2869183Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2869498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2869768Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2869791Z 
2022-11-23T01:44:15.2870055Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2870170Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2870189Z 
2022-11-23T01:44:15.2870299Z OK (skipped=1)
2022-11-23T01:44:15.2870318Z 
2022-11-23T01:44:15.2870423Z Generating XML reports...
2022-11-23T01:44:15.2870871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012505.xml
2022-11-23T01:44:15.2871243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2871424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2871803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2871995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2872015Z 
2022-11-23T01:44:15.2872183Z Running tests...
2022-11-23T01:44:15.2872449Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2872744Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2873008Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2873028Z 
2022-11-23T01:44:15.2873291Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2873406Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2873425Z 
2022-11-23T01:44:15.2873537Z OK (skipped=1)
2022-11-23T01:44:15.2873557Z 
2022-11-23T01:44:15.2873684Z Generating XML reports...
2022-11-23T01:44:15.2874130Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012508.xml
2022-11-23T01:44:15.2874502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2874683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2875045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2875240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2875260Z 
2022-11-23T01:44:15.2875370Z Running tests...
2022-11-23T01:44:15.2875634Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2875993Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2876262Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2876281Z 
2022-11-23T01:44:15.2876546Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2876662Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2876681Z 
2022-11-23T01:44:15.2876790Z OK (skipped=1)
2022-11-23T01:44:15.2876813Z 
2022-11-23T01:44:15.2876919Z Generating XML reports...
2022-11-23T01:44:15.2877370Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012510.xml
2022-11-23T01:44:15.2877742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2877919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2878305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2878501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2878521Z 
2022-11-23T01:44:15.2878633Z Running tests...
2022-11-23T01:44:15.2878898Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2879191Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2879477Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-11-23T01:44:15.2879497Z 
2022-11-23T01:44:15.2879758Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2879871Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2879890Z 
2022-11-23T01:44:15.2880000Z OK (skipped=1)
2022-11-23T01:44:15.2880018Z 
2022-11-23T01:44:15.2880142Z Generating XML reports...
2022-11-23T01:44:15.2880594Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012512.xml
2022-11-23T01:44:15.2880965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2881143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2881505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2881792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2881812Z 
2022-11-23T01:44:15.2881920Z Running tests...
2022-11-23T01:44:15.2882186Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2882502Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2882796Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-11-23T01:44:15.2882816Z 
2022-11-23T01:44:15.2883081Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2883194Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2883214Z 
2022-11-23T01:44:15.2883321Z OK (skipped=1)
2022-11-23T01:44:15.2883340Z 
2022-11-23T01:44:15.2883445Z Generating XML reports...
2022-11-23T01:44:15.2883891Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012515.xml
2022-11-23T01:44:15.2884263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2884440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2884817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2885009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2885028Z 
2022-11-23T01:44:15.2885139Z Running tests...
2022-11-23T01:44:15.2885446Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2885767Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2886020Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2886040Z 
2022-11-23T01:44:15.2886305Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2886419Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2886438Z 
2022-11-23T01:44:15.2886546Z OK (skipped=1)
2022-11-23T01:44:15.2886565Z 
2022-11-23T01:44:15.2886689Z Generating XML reports...
2022-11-23T01:44:15.2887132Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012517.xml
2022-11-23T01:44:15.2887501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2887681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2888058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2888230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2888250Z 
2022-11-23T01:44:15.2888360Z Running tests...
2022-11-23T01:44:15.2888628Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2888941Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2889209Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2889431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6158
2022-11-23T01:44:15.2889649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6159
2022-11-23T01:44:15.2890025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2890183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2890564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2890755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2891186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2891363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2891981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2892173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2892428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2892677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2893064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2893466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2893704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2893936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2894039Z ok (5.765s)
2022-11-23T01:44:15.2894060Z 
2022-11-23T01:44:15.2894328Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2894444Z Ran 1 test in 5.765s
2022-11-23T01:44:15.2894464Z 
2022-11-23T01:44:15.2894557Z OK
2022-11-23T01:44:15.2894577Z 
2022-11-23T01:44:15.2894759Z Generating XML reports...
2022-11-23T01:44:15.2895230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012520.xml
2022-11-23T01:44:15.2895602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2895780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2896163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2896355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2896375Z 
2022-11-23T01:44:15.2896485Z Running tests...
2022-11-23T01:44:15.2896749Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2897068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2897318Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2897338Z 
2022-11-23T01:44:15.2897598Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2897711Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2897730Z 
2022-11-23T01:44:15.2897839Z OK (skipped=1)
2022-11-23T01:44:15.2897858Z 
2022-11-23T01:44:15.2897988Z Generating XML reports...
2022-11-23T01:44:15.2898438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012528.xml
2022-11-23T01:44:15.2898808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2898986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2899368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2899545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2899565Z 
2022-11-23T01:44:15.2899675Z Running tests...
2022-11-23T01:44:15.2899943Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2900258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2900526Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2900631Z 
2022-11-23T01:44:15.2900902Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2901017Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2901036Z 
2022-11-23T01:44:15.2901147Z OK (skipped=1)
2022-11-23T01:44:15.2901166Z 
2022-11-23T01:44:15.2901271Z Generating XML reports...
2022-11-23T01:44:15.2901723Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012530.xml
2022-11-23T01:44:15.2902096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2902274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2902652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2902845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2902867Z 
2022-11-23T01:44:15.2902976Z Running tests...
2022-11-23T01:44:15.2903241Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2903556Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2903811Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.2903849Z 
2022-11-23T01:44:15.2904093Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2904256Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2904278Z 
2022-11-23T01:44:15.2904390Z OK (skipped=1)
2022-11-23T01:44:15.2904409Z 
2022-11-23T01:44:15.2904535Z Generating XML reports...
2022-11-23T01:44:15.2904990Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012533.xml
2022-11-23T01:44:15.2905369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2905547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2905929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2906103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2906123Z 
2022-11-23T01:44:15.2906231Z Running tests...
2022-11-23T01:44:15.2906498Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2906812Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2907073Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2907290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6372
2022-11-23T01:44:15.2907513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6373
2022-11-23T01:44:15.2907886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2908044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2908421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2908613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2908979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2909150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2909522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2909714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2910016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2910267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2910654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2911048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2911283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2911516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2911845Z STAGE:2022-11-23 01:25:40 6372:6372 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2912172Z STAGE:2022-11-23 01:25:40 6373:6373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2912509Z STAGE:2022-11-23 01:25:40 6373:6373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2912841Z STAGE:2022-11-23 01:25:40 6372:6372 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2913182Z STAGE:2022-11-23 01:25:40 6373:6373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2913507Z STAGE:2022-11-23 01:25:40 6372:6372 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2913872Z STAGE:2022-11-23 01:25:40 6372:6372 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2914205Z STAGE:2022-11-23 01:25:40 6373:6373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2914535Z STAGE:2022-11-23 01:25:40 6372:6372 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2914874Z STAGE:2022-11-23 01:25:40 6372:6372 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2915205Z STAGE:2022-11-23 01:25:40 6373:6373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2915542Z STAGE:2022-11-23 01:25:40 6373:6373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2915862Z STAGE:2022-11-23 01:25:40 6372:6372 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2916179Z STAGE:2022-11-23 01:25:40 6373:6373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2916496Z STAGE:2022-11-23 01:25:40 6372:6372 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2916829Z STAGE:2022-11-23 01:25:40 6372:6372 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2917155Z STAGE:2022-11-23 01:25:40 6373:6373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2917495Z STAGE:2022-11-23 01:25:40 6373:6373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2917596Z ok (5.622s)
2022-11-23T01:44:15.2917616Z 
2022-11-23T01:44:15.2917877Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2917987Z Ran 1 test in 5.622s
2022-11-23T01:44:15.2918007Z 
2022-11-23T01:44:15.2918097Z OK
2022-11-23T01:44:15.2918116Z 
2022-11-23T01:44:15.2918222Z Generating XML reports...
2022-11-23T01:44:15.2918672Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012535.xml
2022-11-23T01:44:15.2919044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2919221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2919597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2919841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2919861Z 
2022-11-23T01:44:15.2919966Z Running tests...
2022-11-23T01:44:15.2920220Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2920531Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2920781Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2920994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6490
2022-11-23T01:44:15.2921202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6491
2022-11-23T01:44:15.2921563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2921728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2922096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2922279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2922630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2922787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2923150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2923375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2923619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2923860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2924254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2924650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2924870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2925087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2925401Z STAGE:2022-11-23 01:25:48 6490:6490 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2925719Z STAGE:2022-11-23 01:25:48 6491:6491 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2926040Z STAGE:2022-11-23 01:25:48 6491:6491 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2926361Z STAGE:2022-11-23 01:25:48 6490:6490 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2926693Z STAGE:2022-11-23 01:25:48 6491:6491 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2927030Z STAGE:2022-11-23 01:25:48 6490:6490 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2927340Z STAGE:2022-11-23 01:25:48 6490:6490 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2927652Z STAGE:2022-11-23 01:25:48 6491:6491 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2928024Z STAGE:2022-11-23 01:25:48 6490:6490 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2928355Z STAGE:2022-11-23 01:25:48 6490:6490 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2928674Z STAGE:2022-11-23 01:25:48 6491:6491 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2929002Z STAGE:2022-11-23 01:25:48 6491:6491 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2929369Z STAGE:2022-11-23 01:25:48 6490:6490 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2929680Z STAGE:2022-11-23 01:25:48 6491:6491 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2930007Z STAGE:2022-11-23 01:25:48 6490:6490 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2930331Z STAGE:2022-11-23 01:25:48 6491:6491 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2930663Z STAGE:2022-11-23 01:25:48 6490:6490 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2930997Z STAGE:2022-11-23 01:25:48 6491:6491 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2931083Z ok (5.625s)
2022-11-23T01:44:15.2931103Z 
2022-11-23T01:44:15.2931368Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2931710Z Ran 1 test in 5.625s
2022-11-23T01:44:15.2931732Z 
2022-11-23T01:44:15.2931832Z OK
2022-11-23T01:44:15.2931852Z 
2022-11-23T01:44:15.2931977Z Generating XML reports...
2022-11-23T01:44:15.2932436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012543.xml
2022-11-23T01:44:15.2932803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2932977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2933419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2933624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2933644Z 
2022-11-23T01:44:15.2933751Z Running tests...
2022-11-23T01:44:15.2934018Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2934331Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2934609Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2934828Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6608
2022-11-23T01:44:15.2935040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6609
2022-11-23T01:44:15.2935404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2935565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2935937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2936120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2936474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2936642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2937003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2937184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2937422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2937650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2938041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2938439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2938670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2938973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2939309Z STAGE:2022-11-23 01:25:56 6608:6608 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2939636Z STAGE:2022-11-23 01:25:56 6609:6609 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2939968Z STAGE:2022-11-23 01:25:56 6609:6609 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2940302Z STAGE:2022-11-23 01:25:56 6608:6608 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2940629Z STAGE:2022-11-23 01:25:56 6609:6609 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2940965Z STAGE:2022-11-23 01:25:56 6608:6608 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2941276Z STAGE:2022-11-23 01:25:57 6608:6608 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2941589Z STAGE:2022-11-23 01:25:57 6609:6609 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2941906Z STAGE:2022-11-23 01:25:57 6608:6608 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2942234Z STAGE:2022-11-23 01:25:57 6608:6608 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2942552Z STAGE:2022-11-23 01:25:57 6609:6609 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2942927Z STAGE:2022-11-23 01:25:57 6609:6609 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2943248Z STAGE:2022-11-23 01:25:57 6608:6608 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2943554Z STAGE:2022-11-23 01:25:57 6609:6609 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.2943872Z STAGE:2022-11-23 01:25:57 6608:6608 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2944206Z STAGE:2022-11-23 01:25:57 6608:6608 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2944521Z STAGE:2022-11-23 01:25:57 6609:6609 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.2944846Z STAGE:2022-11-23 01:25:57 6609:6609 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.2944938Z ok (5.634s)
2022-11-23T01:44:15.2944958Z 
2022-11-23T01:44:15.2945212Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2945314Z Ran 1 test in 5.634s
2022-11-23T01:44:15.2945336Z 
2022-11-23T01:44:15.2945411Z OK
2022-11-23T01:44:15.2945436Z 
2022-11-23T01:44:15.2945543Z Generating XML reports...
2022-11-23T01:44:15.2945983Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012551.xml
2022-11-23T01:44:15.2946346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2946516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2946885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2947067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2947087Z 
2022-11-23T01:44:15.2947185Z Running tests...
2022-11-23T01:44:15.2947437Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2947735Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2947971Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:44:15.2947990Z 
2022-11-23T01:44:15.2948253Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2948370Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2948443Z 
2022-11-23T01:44:15.2948556Z OK (skipped=1)
2022-11-23T01:44:15.2948574Z 
2022-11-23T01:44:15.2948698Z Generating XML reports...
2022-11-23T01:44:15.2949135Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012600.xml
2022-11-23T01:44:15.2949495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2949661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2950026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2950208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2950227Z 
2022-11-23T01:44:15.2950325Z Running tests...
2022-11-23T01:44:15.2950576Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2950876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2951126Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:44:15.2951146Z 
2022-11-23T01:44:15.2951396Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2951498Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2951518Z 
2022-11-23T01:44:15.2951608Z OK (skipped=1)
2022-11-23T01:44:15.2951633Z 
2022-11-23T01:44:15.2951740Z Generating XML reports...
2022-11-23T01:44:15.2952232Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012602.xml
2022-11-23T01:44:15.2952606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2952771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2953140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2953324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2953344Z 
2022-11-23T01:44:15.2953441Z Running tests...
2022-11-23T01:44:15.2953693Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2953988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2954231Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2954443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6792
2022-11-23T01:44:15.2954650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6793
2022-11-23T01:44:15.2955008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2955173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2955546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2955727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2956071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2956234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2956601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2956782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2957018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2957253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2957707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2958092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2958312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2958526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2958618Z ok (5.567s)
2022-11-23T01:44:15.2958637Z 
2022-11-23T01:44:15.2958893Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2958996Z Ran 1 test in 5.567s
2022-11-23T01:44:15.2959015Z 
2022-11-23T01:44:15.2959097Z OK
2022-11-23T01:44:15.2959115Z 
2022-11-23T01:44:15.2959228Z Generating XML reports...
2022-11-23T01:44:15.2959664Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012604.xml
2022-11-23T01:44:15.2960027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2960193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2960555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2960736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2960755Z 
2022-11-23T01:44:15.2960852Z Running tests...
2022-11-23T01:44:15.2961142Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2961452Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2961708Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2961918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6909
2022-11-23T01:44:15.2962128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6910
2022-11-23T01:44:15.2962480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2962646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2963010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2963188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2963544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2963709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2964068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2964250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2964485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2964712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2965101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2965490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2965707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2965925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2966794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py:853: UserWarning: Specified kernel cache directory could not be created! This disables kernel caching. Specified directory is /var/lib/jenkins/.cache/torch/kernels. This warning will appear only once per process. (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cuda/jit_utils.cpp:1444.)
2022-11-23T01:44:15.2967063Z   matches = torch.isclose(actual, expected, rtol=rtol, atol=atol, equal_nan=equal_nan)
2022-11-23T01:44:15.2967155Z ok (6.066s)
2022-11-23T01:44:15.2967174Z 
2022-11-23T01:44:15.2967430Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2967526Z Ran 1 test in 6.067s
2022-11-23T01:44:15.2967555Z 
2022-11-23T01:44:15.2967631Z OK
2022-11-23T01:44:15.2967650Z 
2022-11-23T01:44:15.2967763Z Generating XML reports...
2022-11-23T01:44:15.2968200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012612.xml
2022-11-23T01:44:15.2968560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2968729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2969097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2969286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2969306Z 
2022-11-23T01:44:15.2969416Z Running tests...
2022-11-23T01:44:15.2969661Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2970024Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2970283Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:44:15.2970303Z 
2022-11-23T01:44:15.2970557Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2970673Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2970696Z 
2022-11-23T01:44:15.2970806Z OK (skipped=1)
2022-11-23T01:44:15.2970825Z 
2022-11-23T01:44:15.2970939Z Generating XML reports...
2022-11-23T01:44:15.2971382Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012621.xml
2022-11-23T01:44:15.2971938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2972101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2972483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2972674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2972693Z 
2022-11-23T01:44:15.2972792Z Running tests...
2022-11-23T01:44:15.2973044Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2973348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2973608Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2973816Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7059
2022-11-23T01:44:15.2974016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7060
2022-11-23T01:44:15.2974374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2974546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2974910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2975090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2975440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2975689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2976058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2976236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2976465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2976701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2977094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2977480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2977711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2977949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.2978172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2978399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.2978789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.2979229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.2979339Z ok (5.666s)
2022-11-23T01:44:15.2979359Z 
2022-11-23T01:44:15.2979615Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2979725Z Ran 1 test in 5.666s
2022-11-23T01:44:15.2979745Z 
2022-11-23T01:44:15.2979839Z OK
2022-11-23T01:44:15.2979861Z 
2022-11-23T01:44:15.2979977Z Generating XML reports...
2022-11-23T01:44:15.2980424Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012623.xml
2022-11-23T01:44:15.2980786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2980961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2981325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2981510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2981529Z 
2022-11-23T01:44:15.2981636Z Running tests...
2022-11-23T01:44:15.2981888Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2982199Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2982449Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:44:15.2982469Z 
2022-11-23T01:44:15.2982729Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2982831Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2982850Z 
2022-11-23T01:44:15.2982941Z OK (skipped=1)
2022-11-23T01:44:15.2982974Z 
2022-11-23T01:44:15.2983081Z Generating XML reports...
2022-11-23T01:44:15.2983520Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012632.xml
2022-11-23T01:44:15.2983893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2984061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2984434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2984671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2984691Z 
2022-11-23T01:44:15.2984798Z Running tests...
2022-11-23T01:44:15.2985055Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2985349Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2985601Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.2985821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7211
2022-11-23T01:44:15.2986033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7212
2022-11-23T01:44:15.2986398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2986563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2986941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2987127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2987473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2987644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2988009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2988243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2988486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.2988720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.2989120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2989510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.2989739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.2989952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.2990103Z skip: Skipped due to small world size. (4.264s)
2022-11-23T01:44:15.2990122Z 
2022-11-23T01:44:15.2990387Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2990492Z Ran 1 test in 4.264s
2022-11-23T01:44:15.2990511Z 
2022-11-23T01:44:15.2990614Z OK (skipped=1)
2022-11-23T01:44:15.2990633Z 
2022-11-23T01:44:15.2990748Z Generating XML reports...
2022-11-23T01:44:15.2991195Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012634.xml
2022-11-23T01:44:15.2991558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2991725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2992087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2992276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2992295Z 
2022-11-23T01:44:15.2992407Z Running tests...
2022-11-23T01:44:15.2992664Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2992967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2993252Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:44:15.2993272Z 
2022-11-23T01:44:15.2993524Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2993687Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2993706Z 
2022-11-23T01:44:15.2993803Z OK (skipped=1)
2022-11-23T01:44:15.2993823Z 
2022-11-23T01:44:15.2993929Z Generating XML reports...
2022-11-23T01:44:15.2994377Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012641.xml
2022-11-23T01:44:15.2994749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2994920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2995292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2995472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2995491Z 
2022-11-23T01:44:15.2995597Z Running tests...
2022-11-23T01:44:15.2995850Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2996149Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2996444Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:44:15.2996464Z 
2022-11-23T01:44:15.2996714Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2996825Z Ran 1 test in 0.002s
2022-11-23T01:44:15.2996844Z 
2022-11-23T01:44:15.2996943Z OK (skipped=1)
2022-11-23T01:44:15.2996962Z 
2022-11-23T01:44:15.2997127Z Generating XML reports...
2022-11-23T01:44:15.2997576Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012643.xml
2022-11-23T01:44:15.2997943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.2998108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.2998475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.2998662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.2998681Z 
2022-11-23T01:44:15.2998781Z Running tests...
2022-11-23T01:44:15.2999042Z ----------------------------------------------------------------------
2022-11-23T01:44:15.2999344Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.2999622Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3000371Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-11-23T01:44:15.3000395Z 
2022-11-23T01:44:15.3000652Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3000755Z Ran 1 test in 1.641s
2022-11-23T01:44:15.3000774Z 
2022-11-23T01:44:15.3000880Z OK (skipped=1)
2022-11-23T01:44:15.3000899Z 
2022-11-23T01:44:15.3001006Z Generating XML reports...
2022-11-23T01:44:15.3001455Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012646.xml
2022-11-23T01:44:15.3001822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3001994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3002376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3002570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3002641Z 
2022-11-23T01:44:15.3002745Z Running tests...
2022-11-23T01:44:15.3003009Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3003305Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3003587Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3003805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7416
2022-11-23T01:44:15.3004028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7417
2022-11-23T01:44:15.3004386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3004557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3004937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3005120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3005479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3005635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3006010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3006251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3006506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3006753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3007157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3007548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3007775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3008008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3008325Z STAGE:2022-11-23 01:26:55 7417:7417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3008657Z STAGE:2022-11-23 01:26:55 7416:7416 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3008991Z STAGE:2022-11-23 01:26:55 7416:7416 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3009325Z STAGE:2022-11-23 01:26:55 7417:7417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3009667Z STAGE:2022-11-23 01:26:55 7417:7417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3010009Z STAGE:2022-11-23 01:26:55 7416:7416 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3010116Z ok (6.356s)
2022-11-23T01:44:15.3010136Z 
2022-11-23T01:44:15.3010401Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3010497Z Ran 1 test in 6.357s
2022-11-23T01:44:15.3010525Z 
2022-11-23T01:44:15.3010601Z OK
2022-11-23T01:44:15.3010620Z 
2022-11-23T01:44:15.3010740Z Generating XML reports...
2022-11-23T01:44:15.3011192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012650.xml
2022-11-23T01:44:15.3011736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3011920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3012312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3012600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3012619Z 
2022-11-23T01:44:15.3012730Z Running tests...
2022-11-23T01:44:15.3012980Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3013295Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3013595Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:44:15.3013618Z 
2022-11-23T01:44:15.3013881Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3013983Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3014002Z 
2022-11-23T01:44:15.3014106Z OK (skipped=1)
2022-11-23T01:44:15.3014125Z 
2022-11-23T01:44:15.3014250Z Generating XML reports...
2022-11-23T01:44:15.3014699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012659.xml
2022-11-23T01:44:15.3015066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3015226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3015603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3015788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3015808Z 
2022-11-23T01:44:15.3015975Z Running tests...
2022-11-23T01:44:15.3016250Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3016553Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3016843Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3017585Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s)
2022-11-23T01:44:15.3017605Z 
2022-11-23T01:44:15.3017862Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3017957Z Ran 1 test in 1.658s
2022-11-23T01:44:15.3017984Z 
2022-11-23T01:44:15.3018074Z OK (skipped=1)
2022-11-23T01:44:15.3018097Z 
2022-11-23T01:44:15.3018219Z Generating XML reports...
2022-11-23T01:44:15.3018672Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012701.xml
2022-11-23T01:44:15.3019030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3019209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3019580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3019772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3019792Z 
2022-11-23T01:44:15.3019892Z Running tests...
2022-11-23T01:44:15.3020141Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3020451Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3020738Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:44:15.3020758Z 
2022-11-23T01:44:15.3021018Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3021129Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3021148Z 
2022-11-23T01:44:15.3021247Z OK (skipped=1)
2022-11-23T01:44:15.3021320Z 
2022-11-23T01:44:15.3021449Z Generating XML reports...
2022-11-23T01:44:15.3021888Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012705.xml
2022-11-23T01:44:15.3022257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3022416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3022802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3022983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3023002Z 
2022-11-23T01:44:15.3023107Z Running tests...
2022-11-23T01:44:15.3023362Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3023675Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3023956Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3024172Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7637
2022-11-23T01:44:15.3024373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7638
2022-11-23T01:44:15.3024734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3024906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3025366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3025562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3025917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3026092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3026467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3026648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3026879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3027123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3027519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3027962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3028197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3028420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3028576Z skip: Skipped due to small world size. (4.361s)
2022-11-23T01:44:15.3028596Z 
2022-11-23T01:44:15.3028853Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3028963Z Ran 1 test in 4.361s
2022-11-23T01:44:15.3028983Z 
2022-11-23T01:44:15.3029072Z OK (skipped=1)
2022-11-23T01:44:15.3029091Z 
2022-11-23T01:44:15.3029206Z Generating XML reports...
2022-11-23T01:44:15.3029655Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012708.xml
2022-11-23T01:44:15.3030016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3030189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3030570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3030825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3030845Z 
2022-11-23T01:44:15.3030952Z Running tests...
2022-11-23T01:44:15.3031203Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3031519Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3031809Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:44:15.3031833Z 
2022-11-23T01:44:15.3032098Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3032214Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3032233Z 
2022-11-23T01:44:15.3032342Z OK (skipped=1)
2022-11-23T01:44:15.3032361Z 
2022-11-23T01:44:15.3032484Z Generating XML reports...
2022-11-23T01:44:15.3032930Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012715.xml
2022-11-23T01:44:15.3033303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3033463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3033843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3034035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3034055Z 
2022-11-23T01:44:15.3034212Z Running tests...
2022-11-23T01:44:15.3034482Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3034787Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3035085Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:44:15.3035108Z 
2022-11-23T01:44:15.3035368Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3035472Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3035491Z 
2022-11-23T01:44:15.3035580Z OK (skipped=1)
2022-11-23T01:44:15.3035613Z 
2022-11-23T01:44:15.3035719Z Generating XML reports...
2022-11-23T01:44:15.3036168Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012717.xml
2022-11-23T01:44:15.3036541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3036708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3037084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3037279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3037302Z 
2022-11-23T01:44:15.3037411Z Running tests...
2022-11-23T01:44:15.3037727Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3038027Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3038306Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3038516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7808
2022-11-23T01:44:15.3038731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7809
2022-11-23T01:44:15.3039102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3039268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3039641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3039891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3040240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3040405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3040777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3040956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3041204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3041442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3041838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3042240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3042462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3042674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3042775Z ok (5.765s)
2022-11-23T01:44:15.3042794Z 
2022-11-23T01:44:15.3043049Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3043159Z Ran 1 test in 5.765s
2022-11-23T01:44:15.3043178Z 
2022-11-23T01:44:15.3043304Z OK
2022-11-23T01:44:15.3043325Z 
2022-11-23T01:44:15.3043452Z Generating XML reports...
2022-11-23T01:44:15.3043892Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012719.xml
2022-11-23T01:44:15.3044258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3044438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3044801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3044984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3045003Z 
2022-11-23T01:44:15.3045109Z Running tests...
2022-11-23T01:44:15.3045361Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3045676Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3045961Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3046167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7925
2022-11-23T01:44:15.3046382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7926
2022-11-23T01:44:15.3046739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3046905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3047281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3047474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3047828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3047998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3048361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3048548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3048795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3049078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3049470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3049854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3050086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3050315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3050408Z ok (5.563s)
2022-11-23T01:44:15.3050428Z 
2022-11-23T01:44:15.3050691Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3050793Z Ran 1 test in 5.563s
2022-11-23T01:44:15.3050813Z 
2022-11-23T01:44:15.3050892Z OK
2022-11-23T01:44:15.3050926Z 
2022-11-23T01:44:15.3051032Z Generating XML reports...
2022-11-23T01:44:15.3051775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012728.xml
2022-11-23T01:44:15.3052161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3052331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3052783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3052978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3052998Z 
2022-11-23T01:44:15.3053109Z Running tests...
2022-11-23T01:44:15.3053374Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3053671Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3053981Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:44:15.3054000Z 
2022-11-23T01:44:15.3054257Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3054370Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3054389Z 
2022-11-23T01:44:15.3054498Z OK (skipped=1)
2022-11-23T01:44:15.3054518Z 
2022-11-23T01:44:15.3054642Z Generating XML reports...
2022-11-23T01:44:15.3055095Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012736.xml
2022-11-23T01:44:15.3055465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3055645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3056006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3056203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3056223Z 
2022-11-23T01:44:15.3056332Z Running tests...
2022-11-23T01:44:15.3056598Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3056912Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3057209Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3057432Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8075
2022-11-23T01:44:15.3057650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8076
2022-11-23T01:44:15.3058004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3058185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3058643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3058833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3059196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3059372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3059748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3059938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3060185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3060416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3060821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3061221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3061452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3061693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3061968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3062212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3062612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3063006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3063095Z ok (5.740s)
2022-11-23T01:44:15.3063115Z 
2022-11-23T01:44:15.3063382Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3063499Z Ran 1 test in 5.740s
2022-11-23T01:44:15.3063519Z 
2022-11-23T01:44:15.3063611Z OK
2022-11-23T01:44:15.3063630Z 
2022-11-23T01:44:15.3063755Z Generating XML reports...
2022-11-23T01:44:15.3064206Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012738.xml
2022-11-23T01:44:15.3064580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3064758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3065140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3065318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3065337Z 
2022-11-23T01:44:15.3065448Z Running tests...
2022-11-23T01:44:15.3065710Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3066029Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3066327Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:44:15.3066347Z 
2022-11-23T01:44:15.3066612Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3066725Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3066745Z 
2022-11-23T01:44:15.3066853Z OK (skipped=1)
2022-11-23T01:44:15.3066872Z 
2022-11-23T01:44:15.3066979Z Generating XML reports...
2022-11-23T01:44:15.3067431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012746.xml
2022-11-23T01:44:15.3067859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3068036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3068413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3068606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3068625Z 
2022-11-23T01:44:15.3068734Z Running tests...
2022-11-23T01:44:15.3069000Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3069313Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3069585Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3069804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8227
2022-11-23T01:44:15.3070025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8228
2022-11-23T01:44:15.3070393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3070570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3070947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3071183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3071556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3071728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3072088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3072280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3072529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3072774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3073175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3073574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3073805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3074036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3074179Z skip: Skipped due to small world size. (4.256s)
2022-11-23T01:44:15.3074221Z 
2022-11-23T01:44:15.3074470Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3074584Z Ran 1 test in 4.256s
2022-11-23T01:44:15.3074603Z 
2022-11-23T01:44:15.3074711Z OK (skipped=1)
2022-11-23T01:44:15.3074730Z 
2022-11-23T01:44:15.3074854Z Generating XML reports...
2022-11-23T01:44:15.3075306Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012749.xml
2022-11-23T01:44:15.3075676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3075859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3076238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3076410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3076448Z 
2022-11-23T01:44:15.3076540Z Running tests...
2022-11-23T01:44:15.3076864Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3077179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3077440Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3077659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8332
2022-11-23T01:44:15.3077880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8333
2022-11-23T01:44:15.3078253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3078412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3078792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3078984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3079349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3079525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3079900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3080095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3080387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3080641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3081025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3081424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3081662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3081891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3082133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3082380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3082783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3083177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3083282Z ok (5.737s)
2022-11-23T01:44:15.3083301Z 
2022-11-23T01:44:15.3083551Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3083668Z Ran 1 test in 5.737s
2022-11-23T01:44:15.3083687Z 
2022-11-23T01:44:15.3083781Z OK
2022-11-23T01:44:15.3083800Z 
2022-11-23T01:44:15.3083924Z Generating XML reports...
2022-11-23T01:44:15.3084376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012756.xml
2022-11-23T01:44:15.3084747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3084926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3085312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3085506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3085526Z 
2022-11-23T01:44:15.3085618Z Running tests...
2022-11-23T01:44:15.3085884Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3086255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3086519Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3086739Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8456
2022-11-23T01:44:15.3086956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8457
2022-11-23T01:44:15.3087327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3087506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3087869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3088061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3088423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3088603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3088978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3089168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3089416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3089708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3090118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3090496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3090733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3090960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3091115Z skip: Need at least 3 CUDA devices (4.340s)
2022-11-23T01:44:15.3091135Z 
2022-11-23T01:44:15.3091400Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3091732Z Ran 1 test in 4.340s
2022-11-23T01:44:15.3091754Z 
2022-11-23T01:44:15.3091865Z OK (skipped=1)
2022-11-23T01:44:15.3091884Z 
2022-11-23T01:44:15.3092012Z Generating XML reports...
2022-11-23T01:44:15.3092457Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012804.xml
2022-11-23T01:44:15.3092826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3093004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3093387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3093578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3093598Z 
2022-11-23T01:44:15.3093708Z Running tests...
2022-11-23T01:44:15.3093971Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3094284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3094542Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-11-23T01:44:15.3094562Z 
2022-11-23T01:44:15.3094803Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3094917Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3094936Z 
2022-11-23T01:44:15.3095044Z OK (skipped=1)
2022-11-23T01:44:15.3095063Z 
2022-11-23T01:44:15.3095187Z Generating XML reports...
2022-11-23T01:44:15.3095740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012811.xml
2022-11-23T01:44:15.3096110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3096287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3096665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3096864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3096884Z 
2022-11-23T01:44:15.3096976Z Running tests...
2022-11-23T01:44:15.3097242Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3097557Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3097809Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T01:44:15.3097831Z 
2022-11-23T01:44:15.3098095Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3098208Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3098228Z 
2022-11-23T01:44:15.3098337Z OK (skipped=1)
2022-11-23T01:44:15.3098356Z 
2022-11-23T01:44:15.3098480Z Generating XML reports...
2022-11-23T01:44:15.3098912Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012813.xml
2022-11-23T01:44:15.3099351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3099539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3099923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3100116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3100139Z 
2022-11-23T01:44:15.3100250Z Running tests...
2022-11-23T01:44:15.3100516Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3100832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3101087Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3101289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8627
2022-11-23T01:44:15.3101510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8628
2022-11-23T01:44:15.3101883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3102061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3102445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3102643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3103008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3103184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3103540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3103731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3103983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3104229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3104632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3105086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3105319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3105549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3105654Z ok (6.448s)
2022-11-23T01:44:15.3105674Z 
2022-11-23T01:44:15.3105920Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3106037Z Ran 1 test in 6.448s
2022-11-23T01:44:15.3106056Z 
2022-11-23T01:44:15.3106154Z OK
2022-11-23T01:44:15.3106174Z 
2022-11-23T01:44:15.3106299Z Generating XML reports...
2022-11-23T01:44:15.3106752Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012815.xml
2022-11-23T01:44:15.3107123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3107305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3107686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3107882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3107902Z 
2022-11-23T01:44:15.3107994Z Running tests...
2022-11-23T01:44:15.3108259Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3108619Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3108894Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T01:44:15.3108913Z 
2022-11-23T01:44:15.3109177Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3109292Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3109311Z 
2022-11-23T01:44:15.3109426Z OK (skipped=1)
2022-11-23T01:44:15.3109445Z 
2022-11-23T01:44:15.3109571Z Generating XML reports...
2022-11-23T01:44:15.3110002Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012824.xml
2022-11-23T01:44:15.3110373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3110551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3110933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3111125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3111144Z 
2022-11-23T01:44:15.3111255Z Running tests...
2022-11-23T01:44:15.3111517Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3111833Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3112106Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3112310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8775
2022-11-23T01:44:15.3112533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8776
2022-11-23T01:44:15.3112902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3113078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3113458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3113651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3114014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3114247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3114622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3114797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3115046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3115291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3115693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3116094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3116326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3116560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3116720Z skip: Skipped due to small world size. (4.243s)
2022-11-23T01:44:15.3116740Z 
2022-11-23T01:44:15.3117006Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3117102Z Ran 1 test in 4.243s
2022-11-23T01:44:15.3117122Z 
2022-11-23T01:44:15.3117231Z OK (skipped=1)
2022-11-23T01:44:15.3117250Z 
2022-11-23T01:44:15.3117375Z Generating XML reports...
2022-11-23T01:44:15.3117872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012827.xml
2022-11-23T01:44:15.3118252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3118431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3118810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3119007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3119026Z 
2022-11-23T01:44:15.3119118Z Running tests...
2022-11-23T01:44:15.3119386Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3119701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3119961Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T01:44:15.3119981Z 
2022-11-23T01:44:15.3120244Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3120358Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3120377Z 
2022-11-23T01:44:15.3120487Z OK (skipped=1)
2022-11-23T01:44:15.3120506Z 
2022-11-23T01:44:15.3120631Z Generating XML reports...
2022-11-23T01:44:15.3121081Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012834.xml
2022-11-23T01:44:15.3121436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3121615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3121995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3122186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3122206Z 
2022-11-23T01:44:15.3122316Z Running tests...
2022-11-23T01:44:15.3122581Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3122896Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3123160Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3123362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8913
2022-11-23T01:44:15.3123637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8914
2022-11-23T01:44:15.3124012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3124189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3124566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3124763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3125128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3125305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3125676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3125853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3126104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3126352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3126756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3127199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3127439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3127670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3127881Z skip: Skipped due to small world size. (4.232s)
2022-11-23T01:44:15.3127904Z 
2022-11-23T01:44:15.3128176Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3128273Z Ran 1 test in 4.232s
2022-11-23T01:44:15.3128292Z 
2022-11-23T01:44:15.3128402Z OK (skipped=1)
2022-11-23T01:44:15.3128421Z 
2022-11-23T01:44:15.3128546Z Generating XML reports...
2022-11-23T01:44:15.3128996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012836.xml
2022-11-23T01:44:15.3129365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3129547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3129932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3130128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3130147Z 
2022-11-23T01:44:15.3130243Z Running tests...
2022-11-23T01:44:15.3130508Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3130823Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3131106Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T01:44:15.3131125Z 
2022-11-23T01:44:15.3131389Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3131719Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3131741Z 
2022-11-23T01:44:15.3131856Z OK (skipped=1)
2022-11-23T01:44:15.3131875Z 
2022-11-23T01:44:15.3132000Z Generating XML reports...
2022-11-23T01:44:15.3132455Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012843.xml
2022-11-23T01:44:15.3132807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3133069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3133455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3133648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3133667Z 
2022-11-23T01:44:15.3133776Z Running tests...
2022-11-23T01:44:15.3134038Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3134354Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3134624Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T01:44:15.3134644Z 
2022-11-23T01:44:15.3134896Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3134991Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3135011Z 
2022-11-23T01:44:15.3135125Z OK (skipped=1)
2022-11-23T01:44:15.3135144Z 
2022-11-23T01:44:15.3135268Z Generating XML reports...
2022-11-23T01:44:15.3135716Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012845.xml
2022-11-23T01:44:15.3136088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3136265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3136711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3136912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3136931Z 
2022-11-23T01:44:15.3137041Z Running tests...
2022-11-23T01:44:15.3137288Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3137600Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3137880Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T01:44:15.3137899Z 
2022-11-23T01:44:15.3138159Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3138271Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3138290Z 
2022-11-23T01:44:15.3138398Z OK (skipped=1)
2022-11-23T01:44:15.3138418Z 
2022-11-23T01:44:15.3138541Z Generating XML reports...
2022-11-23T01:44:15.3138996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012848.xml
2022-11-23T01:44:15.3139350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3139527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3139907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3140104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3140123Z 
2022-11-23T01:44:15.3140232Z Running tests...
2022-11-23T01:44:15.3140494Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3140807Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3141065Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-11-23T01:44:15.3141084Z 
2022-11-23T01:44:15.3141346Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3141441Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3141460Z 
2022-11-23T01:44:15.3141569Z OK (skipped=1)
2022-11-23T01:44:15.3141588Z 
2022-11-23T01:44:15.3141713Z Generating XML reports...
2022-11-23T01:44:15.3142159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012850.xml
2022-11-23T01:44:15.3142586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3142764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3143143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3143338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3143357Z 
2022-11-23T01:44:15.3143468Z Running tests...
2022-11-23T01:44:15.3143719Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3144032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3144297Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-11-23T01:44:15.3144317Z 
2022-11-23T01:44:15.3144578Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3144691Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3144710Z 
2022-11-23T01:44:15.3144822Z OK (skipped=1)
2022-11-23T01:44:15.3144841Z 
2022-11-23T01:44:15.3144965Z Generating XML reports...
2022-11-23T01:44:15.3145410Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012852.xml
2022-11-23T01:44:15.3145762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3145986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3146375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3146567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3146586Z 
2022-11-23T01:44:15.3146694Z Running tests...
2022-11-23T01:44:15.3146962Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3147274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3147558Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3147782Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9183
2022-11-23T01:44:15.3147982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9184
2022-11-23T01:44:15.3148357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3148536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3148916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3149107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3149471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3149641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3150012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3150184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3150437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3150685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3151086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3151484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3151768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3151998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3152239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3152482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3152868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3153264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3153511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:44:15.3153755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:44:15.3154147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3154537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3154640Z ok (4.346s)
2022-11-23T01:44:15.3154660Z 
2022-11-23T01:44:15.3154926Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3155039Z Ran 1 test in 4.346s
2022-11-23T01:44:15.3155103Z 
2022-11-23T01:44:15.3155182Z OK
2022-11-23T01:44:15.3155218Z 
2022-11-23T01:44:15.3155326Z Generating XML reports...
2022-11-23T01:44:15.3155782Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012855.xml
2022-11-23T01:44:15.3156153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3156335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3156715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3156909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3156928Z 
2022-11-23T01:44:15.3157039Z Running tests...
2022-11-23T01:44:15.3157301Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3157602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3157871Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3158092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9296
2022-11-23T01:44:15.3158311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9297
2022-11-23T01:44:15.3158687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3158863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3159241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3159434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3159785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3159961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3160334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3160524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3160836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3161082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3161483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3161879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3162115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3162326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3162431Z ok (5.322s)
2022-11-23T01:44:15.3162451Z 
2022-11-23T01:44:15.3162717Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3162830Z Ran 1 test in 5.322s
2022-11-23T01:44:15.3162852Z 
2022-11-23T01:44:15.3162947Z OK
2022-11-23T01:44:15.3162966Z 
2022-11-23T01:44:15.3163090Z Generating XML reports...
2022-11-23T01:44:15.3163541Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012902.xml
2022-11-23T01:44:15.3163913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3164093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3164502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3164702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3164722Z 
2022-11-23T01:44:15.3164831Z Running tests...
2022-11-23T01:44:15.3165098Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3165413Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3165699Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3165920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9412
2022-11-23T01:44:15.3166138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9413
2022-11-23T01:44:15.3166486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3166668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3167046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3167239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3167602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3167778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3168151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3168341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3168590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3168814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3169220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3169616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3169849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3170168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3170331Z skip: Skipped due to small world size. (4.222s)
2022-11-23T01:44:15.3170350Z 
2022-11-23T01:44:15.3170618Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3170732Z Ran 1 test in 4.223s
2022-11-23T01:44:15.3170751Z 
2022-11-23T01:44:15.3170860Z OK (skipped=1)
2022-11-23T01:44:15.3170880Z 
2022-11-23T01:44:15.3170986Z Generating XML reports...
2022-11-23T01:44:15.3171661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012909.xml
2022-11-23T01:44:15.3172054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3172231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3172612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3172811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3172830Z 
2022-11-23T01:44:15.3172940Z Running tests...
2022-11-23T01:44:15.3173207Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3173503Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3173773Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3174068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9517
2022-11-23T01:44:15.3174294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9518
2022-11-23T01:44:15.3174667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3174844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3175227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3175421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3175784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3175940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3176318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3176507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3176754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3177001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3177406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3177805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3178039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3178252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3178355Z ok (5.153s)
2022-11-23T01:44:15.3178375Z 
2022-11-23T01:44:15.3178643Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3178756Z Ran 1 test in 5.153s
2022-11-23T01:44:15.3178776Z 
2022-11-23T01:44:15.3178869Z OK
2022-11-23T01:44:15.3178889Z 
2022-11-23T01:44:15.3179014Z Generating XML reports...
2022-11-23T01:44:15.3179470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012916.xml
2022-11-23T01:44:15.3179918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3180097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3180461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3180654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3180672Z 
2022-11-23T01:44:15.3180782Z Running tests...
2022-11-23T01:44:15.3181048Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3181365Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3181641Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3181862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9623
2022-11-23T01:44:15.3182084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9624
2022-11-23T01:44:15.3182440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3182619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3182997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3183234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3183608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3183782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3184156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3184354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3184602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3184832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3185233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3185633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3185865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3186094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3186197Z ok (4.337s)
2022-11-23T01:44:15.3186217Z 
2022-11-23T01:44:15.3186487Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3186600Z Ran 1 test in 4.337s
2022-11-23T01:44:15.3186620Z 
2022-11-23T01:44:15.3186713Z OK
2022-11-23T01:44:15.3186732Z 
2022-11-23T01:44:15.3186839Z Generating XML reports...
2022-11-23T01:44:15.3187290Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012924.xml
2022-11-23T01:44:15.3187661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3187842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3188222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3188414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3188433Z 
2022-11-23T01:44:15.3188544Z Running tests...
2022-11-23T01:44:15.3188870Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3189166Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3189450Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3189669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9728
2022-11-23T01:44:15.3189886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9729
2022-11-23T01:44:15.3190262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3190438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3190814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3191006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3191375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3191533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3191909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3192097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3192392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3192646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3193046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3193444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3193679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3193891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3193995Z ok (5.256s)
2022-11-23T01:44:15.3194015Z 
2022-11-23T01:44:15.3194282Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3194398Z Ran 1 test in 5.256s
2022-11-23T01:44:15.3194417Z 
2022-11-23T01:44:15.3194510Z OK
2022-11-23T01:44:15.3194529Z 
2022-11-23T01:44:15.3194657Z Generating XML reports...
2022-11-23T01:44:15.3195110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012931.xml
2022-11-23T01:44:15.3195481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3195660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3196027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3196221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3196240Z 
2022-11-23T01:44:15.3196350Z Running tests...
2022-11-23T01:44:15.3196613Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3196928Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3197205Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3197425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9844
2022-11-23T01:44:15.3197644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9845
2022-11-23T01:44:15.3197996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3198229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3198611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3198803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3199165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3199343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3199718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3199907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3200153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3200385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3200786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3201179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3201410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3201679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3201787Z ok (5.245s)
2022-11-23T01:44:15.3201807Z 
2022-11-23T01:44:15.3202073Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3202186Z Ran 1 test in 5.245s
2022-11-23T01:44:15.3202206Z 
2022-11-23T01:44:15.3202297Z OK
2022-11-23T01:44:15.3202317Z 
2022-11-23T01:44:15.3202423Z Generating XML reports...
2022-11-23T01:44:15.3202876Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012939.xml
2022-11-23T01:44:15.3203248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3203427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3203807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3204005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3204024Z 
2022-11-23T01:44:15.3204134Z Running tests...
2022-11-23T01:44:15.3204397Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3204694Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3204971Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3205192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9958
2022-11-23T01:44:15.3205411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9959
2022-11-23T01:44:15.3205784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3205962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3206345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3206538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3206904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3207060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3207498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3207690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3207936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3208183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3208587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3208983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3209214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3209427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3209533Z ok (4.447s)
2022-11-23T01:44:15.3209553Z 
2022-11-23T01:44:15.3209819Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3209933Z Ran 1 test in 4.448s
2022-11-23T01:44:15.3209953Z 
2022-11-23T01:44:15.3210049Z OK
2022-11-23T01:44:15.3210068Z 
2022-11-23T01:44:15.3210192Z Generating XML reports...
2022-11-23T01:44:15.3210646Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012946.xml
2022-11-23T01:44:15.3211065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3211249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3211832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3212034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3212058Z 
2022-11-23T01:44:15.3212168Z Running tests...
2022-11-23T01:44:15.3212435Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3212751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3213008Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3213028Z 
2022-11-23T01:44:15.3213289Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3213404Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3213426Z 
2022-11-23T01:44:15.3213535Z OK (skipped=1)
2022-11-23T01:44:15.3213555Z 
2022-11-23T01:44:15.3213660Z Generating XML reports...
2022-11-23T01:44:15.3214109Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012953.xml
2022-11-23T01:44:15.3214482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3214662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3215042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3215235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3215254Z 
2022-11-23T01:44:15.3215364Z Running tests...
2022-11-23T01:44:15.3215626Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3215924Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3216180Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3216932Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-11-23T01:44:15.3217034Z 
2022-11-23T01:44:15.3217307Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3217420Z Ran 1 test in 1.640s
2022-11-23T01:44:15.3217440Z 
2022-11-23T01:44:15.3217549Z OK (skipped=1)
2022-11-23T01:44:15.3217567Z 
2022-11-23T01:44:15.3217693Z Generating XML reports...
2022-11-23T01:44:15.3218142Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012956.xml
2022-11-23T01:44:15.3218517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3218694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3219056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3219252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3219272Z 
2022-11-23T01:44:15.3219381Z Running tests...
2022-11-23T01:44:15.3219643Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3219958Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3220230Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3220250Z 
2022-11-23T01:44:15.3220576Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3220698Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3220717Z 
2022-11-23T01:44:15.3220825Z OK (skipped=1)
2022-11-23T01:44:15.3220844Z 
2022-11-23T01:44:15.3220951Z Generating XML reports...
2022-11-23T01:44:15.3221400Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013000.xml
2022-11-23T01:44:15.3221769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3221946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3222325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3222517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3222537Z 
2022-11-23T01:44:15.3222647Z Running tests...
2022-11-23T01:44:15.3222919Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3223216Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3223487Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3223508Z 
2022-11-23T01:44:15.3223768Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3223884Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3223903Z 
2022-11-23T01:44:15.3224012Z OK (skipped=1)
2022-11-23T01:44:15.3224031Z 
2022-11-23T01:44:15.3224154Z Generating XML reports...
2022-11-23T01:44:15.3224601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013002.xml
2022-11-23T01:44:15.3224971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3225149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3225511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3225703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3225722Z 
2022-11-23T01:44:15.3225831Z Running tests...
2022-11-23T01:44:15.3226095Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3226480Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3226749Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-11-23T01:44:15.3226769Z 
2022-11-23T01:44:15.3227029Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3227142Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3227161Z 
2022-11-23T01:44:15.3227273Z OK (skipped=1)
2022-11-23T01:44:15.3227292Z 
2022-11-23T01:44:15.3227401Z Generating XML reports...
2022-11-23T01:44:15.3227853Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013005.xml
2022-11-23T01:44:15.3228273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3228449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3228833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3229026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3229045Z 
2022-11-23T01:44:15.3229155Z Running tests...
2022-11-23T01:44:15.3229418Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3229714Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3230028Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3230785Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.608s)
2022-11-23T01:44:15.3230809Z 
2022-11-23T01:44:15.3231071Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3231186Z Ran 1 test in 1.608s
2022-11-23T01:44:15.3231205Z 
2022-11-23T01:44:15.3231313Z OK (skipped=1)
2022-11-23T01:44:15.3231332Z 
2022-11-23T01:44:15.3231455Z Generating XML reports...
2022-11-23T01:44:15.3231899Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013007.xml
2022-11-23T01:44:15.3232273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3232446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3232808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3233001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3233023Z 
2022-11-23T01:44:15.3233133Z Running tests...
2022-11-23T01:44:15.3233397Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3233711Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3234023Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3234767Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T01:44:15.3234788Z 
2022-11-23T01:44:15.3235050Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3235164Z Ran 1 test in 1.642s
2022-11-23T01:44:15.3235184Z 
2022-11-23T01:44:15.3235292Z OK (skipped=1)
2022-11-23T01:44:15.3235367Z 
2022-11-23T01:44:15.3235479Z Generating XML reports...
2022-11-23T01:44:15.3235930Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013011.xml
2022-11-23T01:44:15.3236300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3236476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3236860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3237052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3237072Z 
2022-11-23T01:44:15.3237182Z Running tests...
2022-11-23T01:44:15.3237447Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3237741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3238065Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3238809Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.629s)
2022-11-23T01:44:15.3238830Z 
2022-11-23T01:44:15.3239136Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3239254Z Ran 1 test in 1.629s
2022-11-23T01:44:15.3239274Z 
2022-11-23T01:44:15.3239384Z OK (skipped=1)
2022-11-23T01:44:15.3239403Z 
2022-11-23T01:44:15.3239530Z Generating XML reports...
2022-11-23T01:44:15.3239979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013015.xml
2022-11-23T01:44:15.3240354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3240530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3240892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3241088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3241107Z 
2022-11-23T01:44:15.3241217Z Running tests...
2022-11-23T01:44:15.3241483Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3241797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3242062Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3242284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10331
2022-11-23T01:44:15.3242510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10332
2022-11-23T01:44:15.3242880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3243038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3243419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3243611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3243983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3244158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3244530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3244719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3245022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3245252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3245656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3246054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3246292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3246523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3246783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3xzpttb
2022-11-23T01:44:15.3247057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3xzpttb/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3247316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fqvftc_
2022-11-23T01:44:15.3247584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fqvftc_/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3247802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3248040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3248141Z ok (5.643s)
2022-11-23T01:44:15.3248206Z 
2022-11-23T01:44:15.3248481Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3248597Z Ran 1 test in 5.643s
2022-11-23T01:44:15.3248616Z 
2022-11-23T01:44:15.3248709Z OK
2022-11-23T01:44:15.3248728Z 
2022-11-23T01:44:15.3248852Z Generating XML reports...
2022-11-23T01:44:15.3249303Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013020.xml
2022-11-23T01:44:15.3249678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3249838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3250220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3250413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3250432Z 
2022-11-23T01:44:15.3250544Z Running tests...
2022-11-23T01:44:15.3250810Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3251126Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3251404Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3251855Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10449
2022-11-23T01:44:15.3252060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10450
2022-11-23T01:44:15.3252439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3252615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3252994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3253190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3253557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3253734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3254109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3254384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3254614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3254861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3255264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3255665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3255896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3256128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3256391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojh219q9
2022-11-23T01:44:15.3256663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojh219q9/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3256900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg5bw1i0x
2022-11-23T01:44:15.3257171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg5bw1i0x/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3257408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3257704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3257948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3258182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3258285Z ok (5.622s)
2022-11-23T01:44:15.3258305Z 
2022-11-23T01:44:15.3258576Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3258693Z Ran 1 test in 5.622s
2022-11-23T01:44:15.3258714Z 
2022-11-23T01:44:15.3258789Z OK
2022-11-23T01:44:15.3258809Z 
2022-11-23T01:44:15.3258934Z Generating XML reports...
2022-11-23T01:44:15.3259388Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013028.xml
2022-11-23T01:44:15.3259760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3259942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3260323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3260517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3260537Z 
2022-11-23T01:44:15.3260648Z Running tests...
2022-11-23T01:44:15.3260893Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3261214Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3261485Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3262232Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T01:44:15.3262255Z 
2022-11-23T01:44:15.3262519Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3262633Z Ran 1 test in 1.642s
2022-11-23T01:44:15.3262652Z 
2022-11-23T01:44:15.3262762Z OK (skipped=1)
2022-11-23T01:44:15.3262781Z 
2022-11-23T01:44:15.3262906Z Generating XML reports...
2022-11-23T01:44:15.3263358Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013036.xml
2022-11-23T01:44:15.3263789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3263949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3264331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3264522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3264541Z 
2022-11-23T01:44:15.3264655Z Running tests...
2022-11-23T01:44:15.3264922Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3265238Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3265530Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3266276Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.606s)
2022-11-23T01:44:15.3266296Z 
2022-11-23T01:44:15.3266558Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3266670Z Ran 1 test in 1.606s
2022-11-23T01:44:15.3266690Z 
2022-11-23T01:44:15.3266780Z OK (skipped=1)
2022-11-23T01:44:15.3266846Z 
2022-11-23T01:44:15.3266979Z Generating XML reports...
2022-11-23T01:44:15.3267430Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013040.xml
2022-11-23T01:44:15.3267800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3267981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3268362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3268557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3268576Z 
2022-11-23T01:44:15.3268686Z Running tests...
2022-11-23T01:44:15.3268930Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3269247Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3269538Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3269759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10635
2022-11-23T01:44:15.3269981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10636
2022-11-23T01:44:15.3270355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3270535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3270913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3271109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3271455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3271635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3272008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3272198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3272446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3272745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3273146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3273544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3273777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3273994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3274254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxqlxt4o
2022-11-23T01:44:15.3274528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxqlxt4o/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3274783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrhjpxaz
2022-11-23T01:44:15.3275059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrhjpxaz/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3275272Z <class 'torch.nn.parameter.Parameter'>
2022-11-23T01:44:15.3275376Z ok (5.635s)
2022-11-23T01:44:15.3275396Z 
2022-11-23T01:44:15.3275663Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3275758Z Ran 1 test in 5.635s
2022-11-23T01:44:15.3275777Z 
2022-11-23T01:44:15.3275871Z OK
2022-11-23T01:44:15.3275891Z 
2022-11-23T01:44:15.3276015Z Generating XML reports...
2022-11-23T01:44:15.3276513Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml
2022-11-23T01:44:15.3276892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3277070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3277454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3277649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3277669Z 
2022-11-23T01:44:15.3277777Z Running tests...
2022-11-23T01:44:15.3278023Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3278338Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3278650Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3278871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10750
2022-11-23T01:44:15.3279092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10751
2022-11-23T01:44:15.3279459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3279640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3280016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3280190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3280549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3280723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3281099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3281289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3281537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3281837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3282239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3282634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3282849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3283082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3283341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdc3b5zw
2022-11-23T01:44:15.3283614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdc3b5zw/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3283869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9_a_fhu
2022-11-23T01:44:15.3284138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9_a_fhu/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3284242Z ok (5.548s)
2022-11-23T01:44:15.3284262Z 
2022-11-23T01:44:15.3284535Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3284631Z Ran 1 test in 5.548s
2022-11-23T01:44:15.3284669Z 
2022-11-23T01:44:15.3284745Z OK
2022-11-23T01:44:15.3284764Z 
2022-11-23T01:44:15.3284886Z Generating XML reports...
2022-11-23T01:44:15.3285386Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013052.xml
2022-11-23T01:44:15.3285761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3285938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3286318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3286514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3286533Z 
2022-11-23T01:44:15.3286643Z Running tests...
2022-11-23T01:44:15.3286890Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3287206Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3287468Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3287690Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10865
2022-11-23T01:44:15.3287910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10866
2022-11-23T01:44:15.3288278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3288455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3288837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3289011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3289373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3289548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3289925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3290113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3290359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3290603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3291074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3291635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3291860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3292090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3292357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6zlc6pv
2022-11-23T01:44:15.3292630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6zlc6pv/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3292888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpleue9c5n
2022-11-23T01:44:15.3293157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpleue9c5n/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3293398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3293636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3293722Z ok (6.126s)
2022-11-23T01:44:15.3293761Z 
2022-11-23T01:44:15.3294013Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3294126Z Ran 1 test in 6.126s
2022-11-23T01:44:15.3294145Z 
2022-11-23T01:44:15.3294239Z OK
2022-11-23T01:44:15.3294258Z 
2022-11-23T01:44:15.3294382Z Generating XML reports...
2022-11-23T01:44:15.3294913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013100.xml
2022-11-23T01:44:15.3295296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3295474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3295857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3296033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3296070Z 
2022-11-23T01:44:15.3296161Z Running tests...
2022-11-23T01:44:15.3296429Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3296746Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3297040Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3297263Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10984
2022-11-23T01:44:15.3297482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10985
2022-11-23T01:44:15.3297854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3298030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3298394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3298585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3298945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3299119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3299495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3299683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3299931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3300177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3300642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3301039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3301271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3301500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3301760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6f74uis4
2022-11-23T01:44:15.3302028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6f74uis4/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3302282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6vyqrgz
2022-11-23T01:44:15.3302555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6vyqrgz/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3303380Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:44:15.3303493Z ok (5.740s)
2022-11-23T01:44:15.3303513Z 
2022-11-23T01:44:15.3303783Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3303879Z Ran 1 test in 5.740s
2022-11-23T01:44:15.3303898Z 
2022-11-23T01:44:15.3303992Z OK
2022-11-23T01:44:15.3304011Z 
2022-11-23T01:44:15.3304137Z Generating XML reports...
2022-11-23T01:44:15.3304594Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013109.xml
2022-11-23T01:44:15.3304964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3305141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3305521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3305718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3305737Z 
2022-11-23T01:44:15.3305829Z Running tests...
2022-11-23T01:44:15.3306093Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3306409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3306691Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3307441Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2022-11-23T01:44:15.3307461Z 
2022-11-23T01:44:15.3307724Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3307838Z Ran 1 test in 1.646s
2022-11-23T01:44:15.3307857Z 
2022-11-23T01:44:15.3307968Z OK (skipped=1)
2022-11-23T01:44:15.3307988Z 
2022-11-23T01:44:15.3308111Z Generating XML reports...
2022-11-23T01:44:15.3308559Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013117.xml
2022-11-23T01:44:15.3308914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3309151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3309533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3309724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3309744Z 
2022-11-23T01:44:15.3309855Z Running tests...
2022-11-23T01:44:15.3310121Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3310439Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3310767Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-11-23T01:44:15.3310787Z 
2022-11-23T01:44:15.3311047Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3311142Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3311161Z 
2022-11-23T01:44:15.3311275Z OK (skipped=1)
2022-11-23T01:44:15.3311294Z 
2022-11-23T01:44:15.3311418Z Generating XML reports...
2022-11-23T01:44:15.3311866Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013122.xml
2022-11-23T01:44:15.3312235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3312412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3312877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3313076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3313096Z 
2022-11-23T01:44:15.3313188Z Running tests...
2022-11-23T01:44:15.3313452Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3313763Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3314016Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3314763Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.613s)
2022-11-23T01:44:15.3314784Z 
2022-11-23T01:44:15.3315046Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3315163Z Ran 1 test in 1.613s
2022-11-23T01:44:15.3315182Z 
2022-11-23T01:44:15.3315291Z OK (skipped=1)
2022-11-23T01:44:15.3315310Z 
2022-11-23T01:44:15.3315435Z Generating XML reports...
2022-11-23T01:44:15.3315883Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013124.xml
2022-11-23T01:44:15.3316237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3316419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3316806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3317000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3317020Z 
2022-11-23T01:44:15.3317128Z Running tests...
2022-11-23T01:44:15.3317394Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3317711Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3317983Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3318186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11203
2022-11-23T01:44:15.3318405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11204
2022-11-23T01:44:15.3318839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3319017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3319395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3319586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3319951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3320125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3320501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3320674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3320925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3321171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3321572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3321969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3322246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3322484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3322745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavx9mc6m
2022-11-23T01:44:15.3323019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavx9mc6m/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3323259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2vb9l1i
2022-11-23T01:44:15.3323529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2vb9l1i/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3324321Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T01:44:15.3324651Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T01:44:15.3325435Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T01:44:15.3325769Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T01:44:15.3325876Z ok (6.104s)
2022-11-23T01:44:15.3325896Z 
2022-11-23T01:44:15.3326160Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3326276Z Ran 1 test in 6.105s
2022-11-23T01:44:15.3326295Z 
2022-11-23T01:44:15.3326388Z OK
2022-11-23T01:44:15.3326407Z 
2022-11-23T01:44:15.3326514Z Generating XML reports...
2022-11-23T01:44:15.3326964Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013128.xml
2022-11-23T01:44:15.3327335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3327514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3328003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3328198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3328218Z 
2022-11-23T01:44:15.3328332Z Running tests...
2022-11-23T01:44:15.3328599Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3328913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3329171Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3329916Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2022-11-23T01:44:15.3329940Z 
2022-11-23T01:44:15.3330203Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3330316Z Ran 1 test in 1.646s
2022-11-23T01:44:15.3330335Z 
2022-11-23T01:44:15.3330442Z OK (skipped=1)
2022-11-23T01:44:15.3330461Z 
2022-11-23T01:44:15.3330584Z Generating XML reports...
2022-11-23T01:44:15.3331032Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013137.xml
2022-11-23T01:44:15.3331608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3331810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3332201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3332376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3332396Z 
2022-11-23T01:44:15.3332511Z Running tests...
2022-11-23T01:44:15.3332775Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3333090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3333362Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3334109Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s)
2022-11-23T01:44:15.3334130Z 
2022-11-23T01:44:15.3334394Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3334509Z Ran 1 test in 1.659s
2022-11-23T01:44:15.3334528Z 
2022-11-23T01:44:15.3334636Z OK (skipped=1)
2022-11-23T01:44:15.3334655Z 
2022-11-23T01:44:15.3334764Z Generating XML reports...
2022-11-23T01:44:15.3335214Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013141.xml
2022-11-23T01:44:15.3335584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3335762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3336143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3336341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3336361Z 
2022-11-23T01:44:15.3336471Z Running tests...
2022-11-23T01:44:15.3336735Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3337049Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3337324Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3337689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11420
2022-11-23T01:44:15.3337911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11421
2022-11-23T01:44:15.3338289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3338466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3338848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3339041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3339402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3339559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3339937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3340126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3340375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3340620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3341085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3341496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3341729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3341974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3342186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3342423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3342822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3343215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3343477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kt_c5hk
2022-11-23T01:44:15.3343746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kt_c5hk/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3344002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxar4uhu7
2022-11-23T01:44:15.3344276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxar4uhu7/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3344512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3344733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3344968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3345197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3345300Z ok (6.371s)
2022-11-23T01:44:15.3345323Z 
2022-11-23T01:44:15.3345593Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3345706Z Ran 1 test in 6.371s
2022-11-23T01:44:15.3345726Z 
2022-11-23T01:44:15.3345819Z OK
2022-11-23T01:44:15.3345838Z 
2022-11-23T01:44:15.3345963Z Generating XML reports...
2022-11-23T01:44:15.3346395Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013145.xml
2022-11-23T01:44:15.3346829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3347007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3347389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3347583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3347602Z 
2022-11-23T01:44:15.3347717Z Running tests...
2022-11-23T01:44:15.3347979Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3348292Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3348569Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3348777Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11541
2022-11-23T01:44:15.3348999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11542
2022-11-23T01:44:15.3349370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3349549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3349934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3350172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3350547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3350721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3351081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3351276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3351524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3351771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3352172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3352575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3352805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3353087Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:44:15.3353312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3353569Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:44:15.3353831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1uvzrp_
2022-11-23T01:44:15.3354106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1uvzrp_/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3354363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1zevapt
2022-11-23T01:44:15.3354636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1zevapt/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3354873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3355108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3355337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3355634Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3355892Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:44:15.3356172Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:44:15.3356450Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:44:15.3356722Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:44:15.3356954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3357190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3357423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3357659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3357933Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:44:15.3358193Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:44:15.3358473Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T01:44:15.3358787Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T01:44:15.3359026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3359259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3359498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3359734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3359838Z ok (6.839s)
2022-11-23T01:44:15.3359857Z 
2022-11-23T01:44:15.3360129Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3360225Z Ran 1 test in 6.839s
2022-11-23T01:44:15.3360244Z 
2022-11-23T01:44:15.3360338Z OK
2022-11-23T01:44:15.3360357Z 
2022-11-23T01:44:15.3360482Z Generating XML reports...
2022-11-23T01:44:15.3360935Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013154.xml
2022-11-23T01:44:15.3361307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3361484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3361867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3362065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3362084Z 
2022-11-23T01:44:15.3362175Z Running tests...
2022-11-23T01:44:15.3362446Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3362759Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3363033Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3363782Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.597s)
2022-11-23T01:44:15.3363802Z 
2022-11-23T01:44:15.3364123Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3364237Z Ran 1 test in 1.598s
2022-11-23T01:44:15.3364257Z 
2022-11-23T01:44:15.3364365Z OK (skipped=1)
2022-11-23T01:44:15.3364384Z 
2022-11-23T01:44:15.3364508Z Generating XML reports...
2022-11-23T01:44:15.3364957Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013203.xml
2022-11-23T01:44:15.3365309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3365491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3365872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3366065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3366084Z 
2022-11-23T01:44:15.3366193Z Running tests...
2022-11-23T01:44:15.3366462Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3366778Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3367052Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3367256Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11694
2022-11-23T01:44:15.3367477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11695
2022-11-23T01:44:15.3367895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3368079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3368461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3368652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3369021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3369196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3369571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3369745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3369994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3370244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3370648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3371045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3371280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3372013Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:44:15.3372254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3372793Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:44:15.3373136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwv3nqm7
2022-11-23T01:44:15.3373408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwv3nqm7/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3373647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvz6du_3n
2022-11-23T01:44:15.3373917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvz6du_3n/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3374153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3374393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3374670Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T01:44:15.3374944Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T01:44:15.3375249Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T01:44:15.3375582Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T01:44:15.3375908Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T01:44:15.3376244Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T01:44:15.3376579Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T01:44:15.3376896Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T01:44:15.3377137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3377368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3377472Z ok (6.222s)
2022-11-23T01:44:15.3377492Z 
2022-11-23T01:44:15.3377770Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3377887Z Ran 1 test in 6.222s
2022-11-23T01:44:15.3377907Z 
2022-11-23T01:44:15.3378001Z OK
2022-11-23T01:44:15.3378020Z 
2022-11-23T01:44:15.3378126Z Generating XML reports...
2022-11-23T01:44:15.3378581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013208.xml
2022-11-23T01:44:15.3378955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3379133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3379519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3379711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3379730Z 
2022-11-23T01:44:15.3379843Z Running tests...
2022-11-23T01:44:15.3380108Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3380403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3380802Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3380822Z 
2022-11-23T01:44:15.3381086Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3381201Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3381220Z 
2022-11-23T01:44:15.3381329Z OK (skipped=1)
2022-11-23T01:44:15.3381398Z 
2022-11-23T01:44:15.3381527Z Generating XML reports...
2022-11-23T01:44:15.3381980Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013216.xml
2022-11-23T01:44:15.3382351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3382527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3382907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3383085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3383105Z 
2022-11-23T01:44:15.3383214Z Running tests...
2022-11-23T01:44:15.3383481Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3383795Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3384191Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3384211Z 
2022-11-23T01:44:15.3384475Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3384588Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3384607Z 
2022-11-23T01:44:15.3384716Z OK (skipped=1)
2022-11-23T01:44:15.3384735Z 
2022-11-23T01:44:15.3384859Z Generating XML reports...
2022-11-23T01:44:15.3385339Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013219.xml
2022-11-23T01:44:15.3385723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3385900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3386280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3386475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3386495Z 
2022-11-23T01:44:15.3386605Z Running tests...
2022-11-23T01:44:15.3386869Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3387184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3387643Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3387664Z 
2022-11-23T01:44:15.3387907Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3388022Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3388041Z 
2022-11-23T01:44:15.3388150Z OK (skipped=1)
2022-11-23T01:44:15.3388173Z 
2022-11-23T01:44:15.3388298Z Generating XML reports...
2022-11-23T01:44:15.3388751Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013221.xml
2022-11-23T01:44:15.3389120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3389301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3389683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3389878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3389898Z 
2022-11-23T01:44:15.3389989Z Running tests...
2022-11-23T01:44:15.3390254Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3390570Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3391069Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3391090Z 
2022-11-23T01:44:15.3391353Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3391466Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3391485Z 
2022-11-23T01:44:15.3391597Z OK (skipped=1)
2022-11-23T01:44:15.3391616Z 
2022-11-23T01:44:15.3391740Z Generating XML reports...
2022-11-23T01:44:15.3392193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013223.xml
2022-11-23T01:44:15.3392547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3392724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3393105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3393297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3393316Z 
2022-11-23T01:44:15.3393425Z Running tests...
2022-11-23T01:44:15.3393691Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3394006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3394509Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3394531Z 
2022-11-23T01:44:15.3394797Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3394893Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3394931Z 
2022-11-23T01:44:15.3395027Z OK (skipped=1)
2022-11-23T01:44:15.3395045Z 
2022-11-23T01:44:15.3395172Z Generating XML reports...
2022-11-23T01:44:15.3395624Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013226.xml
2022-11-23T01:44:15.3395995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3396171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3396552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3396749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3396768Z 
2022-11-23T01:44:15.3396878Z Running tests...
2022-11-23T01:44:15.3397124Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3397436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3397884Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3397904Z 
2022-11-23T01:44:15.3398165Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3398279Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3398298Z 
2022-11-23T01:44:15.3398406Z OK (skipped=1)
2022-11-23T01:44:15.3398425Z 
2022-11-23T01:44:15.3398552Z Generating XML reports...
2022-11-23T01:44:15.3399004Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013228.xml
2022-11-23T01:44:15.3399375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3399553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3399974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3400173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3400192Z 
2022-11-23T01:44:15.3400303Z Running tests...
2022-11-23T01:44:15.3400566Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3400879Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3401318Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3401339Z 
2022-11-23T01:44:15.3401593Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3401705Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3401728Z 
2022-11-23T01:44:15.3401840Z OK (skipped=1)
2022-11-23T01:44:15.3401859Z 
2022-11-23T01:44:15.3401964Z Generating XML reports...
2022-11-23T01:44:15.3402415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013231.xml
2022-11-23T01:44:15.3402786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3402963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3403388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3403589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3403610Z 
2022-11-23T01:44:15.3403720Z Running tests...
2022-11-23T01:44:15.3403987Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3404289Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3404734Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3404774Z 
2022-11-23T01:44:15.3405017Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3405131Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3405151Z 
2022-11-23T01:44:15.3405259Z OK (skipped=1)
2022-11-23T01:44:15.3405282Z 
2022-11-23T01:44:15.3405408Z Generating XML reports...
2022-11-23T01:44:15.3405856Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013233.xml
2022-11-23T01:44:15.3406228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3406410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3406792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3406967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3406985Z 
2022-11-23T01:44:15.3407095Z Running tests...
2022-11-23T01:44:15.3407360Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3407674Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3408122Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3408142Z 
2022-11-23T01:44:15.3408405Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3408574Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3408594Z 
2022-11-23T01:44:15.3408703Z OK (skipped=1)
2022-11-23T01:44:15.3408722Z 
2022-11-23T01:44:15.3408847Z Generating XML reports...
2022-11-23T01:44:15.3409280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013235.xml
2022-11-23T01:44:15.3409657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3409838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3410221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3410415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3410435Z 
2022-11-23T01:44:15.3410545Z Running tests...
2022-11-23T01:44:15.3410808Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3411125Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3411882Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3411907Z 
2022-11-23T01:44:15.3412181Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3412278Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3412298Z 
2022-11-23T01:44:15.3412490Z OK (skipped=1)
2022-11-23T01:44:15.3412512Z 
2022-11-23T01:44:15.3412646Z Generating XML reports...
2022-11-23T01:44:15.3413101Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013238.xml
2022-11-23T01:44:15.3413471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3413653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3414036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3414229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3414248Z 
2022-11-23T01:44:15.3414339Z Running tests...
2022-11-23T01:44:15.3414604Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3414925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3415319Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3415339Z 
2022-11-23T01:44:15.3415600Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3415719Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3415738Z 
2022-11-23T01:44:15.3415846Z OK (skipped=1)
2022-11-23T01:44:15.3415865Z 
2022-11-23T01:44:15.3415988Z Generating XML reports...
2022-11-23T01:44:15.3416436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013240.xml
2022-11-23T01:44:15.3416788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3416966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3417350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3417543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3417562Z 
2022-11-23T01:44:15.3417670Z Running tests...
2022-11-23T01:44:15.3417933Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3418339Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3418727Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T01:44:15.3418748Z 
2022-11-23T01:44:15.3419009Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3419103Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3419144Z 
2022-11-23T01:44:15.3419235Z OK (skipped=1)
2022-11-23T01:44:15.3419257Z 
2022-11-23T01:44:15.3419383Z Generating XML reports...
2022-11-23T01:44:15.3419833Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013243.xml
2022-11-23T01:44:15.3420202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3420381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3420760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3420952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3420972Z 
2022-11-23T01:44:15.3421080Z Running tests...
2022-11-23T01:44:15.3421323Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3421636Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3421953Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3422709Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.669s)
2022-11-23T01:44:15.3422734Z 
2022-11-23T01:44:15.3422994Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3423108Z Ran 1 test in 1.669s
2022-11-23T01:44:15.3423128Z 
2022-11-23T01:44:15.3423237Z OK (skipped=1)
2022-11-23T01:44:15.3423257Z 
2022-11-23T01:44:15.3423382Z Generating XML reports...
2022-11-23T01:44:15.3425535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013245.xml
2022-11-23T01:44:15.3425909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3426069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3426446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3426639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3426662Z 
2022-11-23T01:44:15.3426771Z Running tests...
2022-11-23T01:44:15.3427032Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3427343Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3427598Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3427820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12243
2022-11-23T01:44:15.3428079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12244
2022-11-23T01:44:15.3428460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3428639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3429017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3429270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3429641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3429816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3430191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3430380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3430613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3430859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3431263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3431662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3431895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3432122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3432379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05h16w2_
2022-11-23T01:44:15.3432693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05h16w2_/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3432955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rj0kqsa
2022-11-23T01:44:15.3433206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rj0kqsa/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3433311Z ok (5.854s)
2022-11-23T01:44:15.3433331Z 
2022-11-23T01:44:15.3433601Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3433717Z Ran 1 test in 5.855s
2022-11-23T01:44:15.3433737Z 
2022-11-23T01:44:15.3433830Z OK
2022-11-23T01:44:15.3433850Z 
2022-11-23T01:44:15.3433973Z Generating XML reports...
2022-11-23T01:44:15.3434425Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013249.xml
2022-11-23T01:44:15.3434797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3434961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3435343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3435538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3435558Z 
2022-11-23T01:44:15.3435666Z Running tests...
2022-11-23T01:44:15.3435931Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3436248Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3436521Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3436741Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12357
2022-11-23T01:44:15.3436957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12358
2022-11-23T01:44:15.3437312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3437489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3437867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3438058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3438479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3438656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3439028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3439215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3439445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3439692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3440097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3440490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3440723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3440952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3441209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjdmpjtz
2022-11-23T01:44:15.3441481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjdmpjtz/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3441735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrx40aq7
2022-11-23T01:44:15.3442033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrx40aq7/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3442279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3442510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3442922Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpcjdmpjtz'>
2022-11-23T01:44:15.3443088Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T01:44:15.3443486Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpxrx40aq7'>
2022-11-23T01:44:15.3443651Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T01:44:15.3443754Z ok (6.176s)
2022-11-23T01:44:15.3443775Z 
2022-11-23T01:44:15.3444040Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3444139Z Ran 1 test in 6.176s
2022-11-23T01:44:15.3444159Z 
2022-11-23T01:44:15.3444252Z OK
2022-11-23T01:44:15.3444271Z 
2022-11-23T01:44:15.3444395Z Generating XML reports...
2022-11-23T01:44:15.3444849Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013258.xml
2022-11-23T01:44:15.3445220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3445402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3445782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3445974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3445994Z 
2022-11-23T01:44:15.3446087Z Running tests...
2022-11-23T01:44:15.3446349Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3446667Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3446946Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-11-23T01:44:15.3446965Z 
2022-11-23T01:44:15.3447232Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3447404Z Ran 1 test in 0.006s
2022-11-23T01:44:15.3447424Z 
2022-11-23T01:44:15.3447533Z OK (skipped=1)
2022-11-23T01:44:15.3447552Z 
2022-11-23T01:44:15.3447676Z Generating XML reports...
2022-11-23T01:44:15.3448127Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013306.xml
2022-11-23T01:44:15.3448481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3448657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3449039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3449228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3449247Z 
2022-11-23T01:44:15.3449354Z Running tests...
2022-11-23T01:44:15.3449614Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3449929Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3450192Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3450395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12509
2022-11-23T01:44:15.3450613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12510
2022-11-23T01:44:15.3450986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3451208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3451870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3452073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3452443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3452621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3452993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3453165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3453415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3453819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3454064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3454462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3454691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3454924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3455185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptma2lgxz
2022-11-23T01:44:15.3455455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptma2lgxz/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3455693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4b7cbtj
2022-11-23T01:44:15.3455964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4b7cbtj/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3456198Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3456433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3456535Z ok (6.167s)
2022-11-23T01:44:15.3456556Z 
2022-11-23T01:44:15.3456969Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3457085Z Ran 1 test in 6.168s
2022-11-23T01:44:15.3457105Z 
2022-11-23T01:44:15.3457200Z OK
2022-11-23T01:44:15.3457219Z 
2022-11-23T01:44:15.3457325Z Generating XML reports...
2022-11-23T01:44:15.3457778Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013309.xml
2022-11-23T01:44:15.3458149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3458333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3458713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3458904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3458923Z 
2022-11-23T01:44:15.3459032Z Running tests...
2022-11-23T01:44:15.3459301Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3459613Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3459883Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3460104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12628
2022-11-23T01:44:15.3460321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12629
2022-11-23T01:44:15.3460755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3460944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3461331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3461524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3461896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3462054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3462428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3462616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3462863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3463106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3463506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3463902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3464134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3464363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3464587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3464829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3465227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3465621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3465860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:44:15.3466101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:44:15.3466555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3466948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3467210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzd09_khe
2022-11-23T01:44:15.3467466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzd09_khe/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3467721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bp_kqeu
2022-11-23T01:44:15.3467988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bp_kqeu/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3468092Z ok (5.346s)
2022-11-23T01:44:15.3468112Z 
2022-11-23T01:44:15.3468378Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3468495Z Ran 1 test in 5.346s
2022-11-23T01:44:15.3468514Z 
2022-11-23T01:44:15.3468609Z OK
2022-11-23T01:44:15.3468628Z 
2022-11-23T01:44:15.3468752Z Generating XML reports...
2022-11-23T01:44:15.3469205Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013318.xml
2022-11-23T01:44:15.3469560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3469736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3470160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3470360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3470379Z 
2022-11-23T01:44:15.3470490Z Running tests...
2022-11-23T01:44:15.3470758Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3471075Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3471359Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3471562Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12750
2022-11-23T01:44:15.3471780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12751
2022-11-23T01:44:15.3472151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3472326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3472702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3472891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3473256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3473430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3473801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3473973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3474220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3474466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3474864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3475258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3475553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3475783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3476027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3476270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3476658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3476894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:44:15.3477286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3477525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:44:15.3477917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3478309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3478565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswca_gw1
2022-11-23T01:44:15.3478836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswca_gw1/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3479138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxn8p7uf
2022-11-23T01:44:15.3479397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxn8p7uf/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3479876Z [W ProcessGroupNCCL.cpp:950] [Rank 1] Found key in store: NCCLABORTEDCOMM:20a2f5ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-11-23T01:44:15.3480354Z [W ProcessGroupNCCL.cpp:950] [Rank 0] Found key in store: NCCLABORTEDCOMM:20a2f5ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-11-23T01:44:15.3480461Z ok (16.064s)
2022-11-23T01:44:15.3480481Z 
2022-11-23T01:44:15.3480755Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3480872Z Ran 1 test in 16.065s
2022-11-23T01:44:15.3480891Z 
2022-11-23T01:44:15.3480984Z OK
2022-11-23T01:44:15.3481003Z 
2022-11-23T01:44:15.3481129Z Generating XML reports...
2022-11-23T01:44:15.3481580Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013325.xml
2022-11-23T01:44:15.3481956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3482131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3482494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3482687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3482707Z 
2022-11-23T01:44:15.3482817Z Running tests...
2022-11-23T01:44:15.3483080Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3483394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3483702Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3483997Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12872
2022-11-23T01:44:15.3484218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12873
2022-11-23T01:44:15.3484576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3484753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3485137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3485333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3485698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3485874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3486245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3486437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3486685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3486914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3487314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3487753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3487994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3488228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3488492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5lpoqst
2022-11-23T01:44:15.3488764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5lpoqst/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3489019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaf_g8fpk
2022-11-23T01:44:15.3489286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaf_g8fpk/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3489372Z ok (6.327s)
2022-11-23T01:44:15.3489392Z 
2022-11-23T01:44:15.3489665Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3489778Z Ran 1 test in 6.327s
2022-11-23T01:44:15.3489798Z 
2022-11-23T01:44:15.3489891Z OK
2022-11-23T01:44:15.3489910Z 
2022-11-23T01:44:15.3490035Z Generating XML reports...
2022-11-23T01:44:15.3490489Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013344.xml
2022-11-23T01:44:15.3490864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3491040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3491402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3491832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3491854Z 
2022-11-23T01:44:15.3491965Z Running tests...
2022-11-23T01:44:15.3492240Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3492552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3492845Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3493066Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12991
2022-11-23T01:44:15.3493369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12992
2022-11-23T01:44:15.3493726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3493903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3494283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3494473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3494837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3495012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3495383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3495574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3495819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3496047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3496446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3496903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3497145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3497374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3497636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsoer2i0
2022-11-23T01:44:15.3497911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsoer2i0/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3498166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxucjb75u
2022-11-23T01:44:15.3498433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxucjb75u/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3498519Z ok (6.211s)
2022-11-23T01:44:15.3498539Z 
2022-11-23T01:44:15.3498809Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3498923Z Ran 1 test in 6.211s
2022-11-23T01:44:15.3498942Z 
2022-11-23T01:44:15.3499038Z OK
2022-11-23T01:44:15.3499057Z 
2022-11-23T01:44:15.3499182Z Generating XML reports...
2022-11-23T01:44:15.3499629Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013353.xml
2022-11-23T01:44:15.3500002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3500182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3500543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3500734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3500753Z 
2022-11-23T01:44:15.3500863Z Running tests...
2022-11-23T01:44:15.3501125Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3501441Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3501697Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3501918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13110
2022-11-23T01:44:15.3502135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13111
2022-11-23T01:44:15.3502568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3502729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3503107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3503301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3503665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3503843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3504217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3504404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3504652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3504884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3505285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3505676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3505907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3506178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3506445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9m6b2inf
2022-11-23T01:44:15.3506717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m6b2inf/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3506973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc6945a0
2022-11-23T01:44:15.3507243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc6945a0/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3507329Z ok (6.139s)
2022-11-23T01:44:15.3507348Z 
2022-11-23T01:44:15.3507619Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3507731Z Ran 1 test in 6.139s
2022-11-23T01:44:15.3507750Z 
2022-11-23T01:44:15.3507842Z OK
2022-11-23T01:44:15.3507862Z 
2022-11-23T01:44:15.3507985Z Generating XML reports...
2022-11-23T01:44:15.3508440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013402.xml
2022-11-23T01:44:15.3508809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3508987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3509368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3509548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3509567Z 
2022-11-23T01:44:15.3509677Z Running tests...
2022-11-23T01:44:15.3509940Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3510253Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3510515Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3510736Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13225
2022-11-23T01:44:15.3510954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13226
2022-11-23T01:44:15.3511325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3511539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3511923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3512115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3512480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3512652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3513026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3513213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3513458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3513703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3514091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3514487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3514717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3514945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3515250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pb88qb0
2022-11-23T01:44:15.3515529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pb88qb0/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3515784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6jfci88
2022-11-23T01:44:15.3516048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6jfci88/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3516830Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:44:15.3517606Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:44:15.3517848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3518084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3518170Z ok (6.145s)
2022-11-23T01:44:15.3518190Z 
2022-11-23T01:44:15.3518458Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3518572Z Ran 1 test in 6.145s
2022-11-23T01:44:15.3518592Z 
2022-11-23T01:44:15.3518686Z OK
2022-11-23T01:44:15.3518706Z 
2022-11-23T01:44:15.3518831Z Generating XML reports...
2022-11-23T01:44:15.3519286Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013410.xml
2022-11-23T01:44:15.3519658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3519889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3520276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3520453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3520472Z 
2022-11-23T01:44:15.3520583Z Running tests...
2022-11-23T01:44:15.3520846Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3521160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3521441Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3522189Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2022-11-23T01:44:15.3522212Z 
2022-11-23T01:44:15.3522476Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3522590Z Ran 1 test in 1.635s
2022-11-23T01:44:15.3522609Z 
2022-11-23T01:44:15.3522720Z OK (skipped=1)
2022-11-23T01:44:15.3522739Z 
2022-11-23T01:44:15.3522847Z Generating XML reports...
2022-11-23T01:44:15.3523300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013419.xml
2022-11-23T01:44:15.3523715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3523900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3524283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3524484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3524504Z 
2022-11-23T01:44:15.3524616Z Running tests...
2022-11-23T01:44:15.3524881Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3525195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3525460Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3526208Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.632s)
2022-11-23T01:44:15.3526228Z 
2022-11-23T01:44:15.3526490Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3526605Z Ran 1 test in 1.632s
2022-11-23T01:44:15.3526624Z 
2022-11-23T01:44:15.3526733Z OK (skipped=1)
2022-11-23T01:44:15.3526752Z 
2022-11-23T01:44:15.3526875Z Generating XML reports...
2022-11-23T01:44:15.3527327Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013423.xml
2022-11-23T01:44:15.3527698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3527876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3528293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3528487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3528506Z 
2022-11-23T01:44:15.3528618Z Running tests...
2022-11-23T01:44:15.3528879Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3529251Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3529528Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3529738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13412
2022-11-23T01:44:15.3529960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13413
2022-11-23T01:44:15.3530333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3530512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3530890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3531079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3531652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3531845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3532205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3532400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3532649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3532970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3533390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3533786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3534018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3534251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3534506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7d9z5aug
2022-11-23T01:44:15.3534760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7d9z5aug/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3535014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3uk51b3
2022-11-23T01:44:15.3535281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3uk51b3/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3535621Z STAGE:2022-11-23 01:34:32 13413:13413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3535944Z STAGE:2022-11-23 01:34:32 13412:13412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3536181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3536421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3536759Z STAGE:2022-11-23 01:34:33 13413:13413 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3537089Z STAGE:2022-11-23 01:34:33 13412:13412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3537420Z STAGE:2022-11-23 01:34:33 13413:13413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3537773Z STAGE:2022-11-23 01:34:33 13412:13412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3538554Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:44:15.3539409Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:44:15.3539747Z STAGE:2022-11-23 01:34:33 13413:13413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3540070Z STAGE:2022-11-23 01:34:33 13412:13412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3540407Z STAGE:2022-11-23 01:34:33 13413:13413 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3540737Z STAGE:2022-11-23 01:34:33 13412:13412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3541084Z STAGE:2022-11-23 01:34:33 13413:13413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3541429Z STAGE:2022-11-23 01:34:33 13412:13412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3541531Z ok (6.159s)
2022-11-23T01:44:15.3541552Z 
2022-11-23T01:44:15.3541868Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3541974Z Ran 1 test in 6.159s
2022-11-23T01:44:15.3542011Z 
2022-11-23T01:44:15.3542088Z OK
2022-11-23T01:44:15.3542107Z 
2022-11-23T01:44:15.3542234Z Generating XML reports...
2022-11-23T01:44:15.3542693Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013427.xml
2022-11-23T01:44:15.3543071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3543247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3543628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3543820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3543840Z 
2022-11-23T01:44:15.3543948Z Running tests...
2022-11-23T01:44:15.3544192Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3544503Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3544768Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3544986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13534
2022-11-23T01:44:15.3545208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13535
2022-11-23T01:44:15.3545576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3545752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3546127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3546304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3546666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3546840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3547217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3547479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3547727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3547971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3548375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3548778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3548994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3549222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3549480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8em5x1ko
2022-11-23T01:44:15.3549753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8em5x1ko/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3550007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqr3jl4vh
2022-11-23T01:44:15.3550276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqr3jl4vh/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3550377Z ok (5.652s)
2022-11-23T01:44:15.3550397Z 
2022-11-23T01:44:15.3550667Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3550778Z Ran 1 test in 5.653s
2022-11-23T01:44:15.3550846Z 
2022-11-23T01:44:15.3550927Z OK
2022-11-23T01:44:15.3550946Z 
2022-11-23T01:44:15.3551073Z Generating XML reports...
2022-11-23T01:44:15.3551527Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013436.xml
2022-11-23T01:44:15.3551900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3552083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3552464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3552655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3552674Z 
2022-11-23T01:44:15.3552784Z Running tests...
2022-11-23T01:44:15.3553032Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3553347Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3553623Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3554366Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2022-11-23T01:44:15.3554390Z 
2022-11-23T01:44:15.3554651Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3554761Z Ran 1 test in 1.627s
2022-11-23T01:44:15.3554781Z 
2022-11-23T01:44:15.3554889Z OK (skipped=1)
2022-11-23T01:44:15.3554908Z 
2022-11-23T01:44:15.3555033Z Generating XML reports...
2022-11-23T01:44:15.3555485Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013444.xml
2022-11-23T01:44:15.3555853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3556013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3556390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3556643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3556662Z 
2022-11-23T01:44:15.3556772Z Running tests...
2022-11-23T01:44:15.3557036Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3557349Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3557633Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3557859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13683
2022-11-23T01:44:15.3558063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13684
2022-11-23T01:44:15.3558437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3558613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3558997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3559188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3559551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3559726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3560100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3560335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3560571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3560820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3561221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3561625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3561856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3562086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3562344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2up83wx
2022-11-23T01:44:15.3562618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2up83wx/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3562871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvk2m4i4
2022-11-23T01:44:15.3563121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvk2m4i4/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3564035Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T01:44:15.3564149Z   warnings.warn(
2022-11-23T01:44:15.3565051Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T01:44:15.3565167Z   warnings.warn(
2022-11-23T01:44:15.3565405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3565687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3565791Z ok (5.750s)
2022-11-23T01:44:15.3565810Z 
2022-11-23T01:44:15.3566082Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3566197Z Ran 1 test in 5.750s
2022-11-23T01:44:15.3566217Z 
2022-11-23T01:44:15.3566292Z OK
2022-11-23T01:44:15.3566311Z 
2022-11-23T01:44:15.3566436Z Generating XML reports...
2022-11-23T01:44:15.3566894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013448.xml
2022-11-23T01:44:15.3567264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3567441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3567820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3568014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3568034Z 
2022-11-23T01:44:15.3568141Z Running tests...
2022-11-23T01:44:15.3568405Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3568703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3568978Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3569769Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2022-11-23T01:44:15.3569792Z 
2022-11-23T01:44:15.3570060Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3570182Z Ran 1 test in 1.646s
2022-11-23T01:44:15.3570201Z 
2022-11-23T01:44:15.3570309Z OK (skipped=1)
2022-11-23T01:44:15.3570328Z 
2022-11-23T01:44:15.3570454Z Generating XML reports...
2022-11-23T01:44:15.3570901Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013457.xml
2022-11-23T01:44:15.3571271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3571660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3572067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3572261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3572281Z 
2022-11-23T01:44:15.3572392Z Running tests...
2022-11-23T01:44:15.3572655Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3572973Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3573246Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3573465Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13835
2022-11-23T01:44:15.3573682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13836
2022-11-23T01:44:15.3574038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3574217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3574593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3574782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3575142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3575402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3575781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3575971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3576203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3576453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3576853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3577248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3577481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3577709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3577963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqki0h2e
2022-11-23T01:44:15.3578233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqki0h2e/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3578488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8wa0nyn
2022-11-23T01:44:15.3578800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8wa0nyn/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3579157Z STAGE:2022-11-23 01:35:06 13836:13836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3579487Z STAGE:2022-11-23 01:35:06 13835:13835 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3580111Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.3580229Z   warnings.warn(
2022-11-23T01:44:15.3580842Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T01:44:15.3580953Z   warnings.warn(
2022-11-23T01:44:15.3581194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3581433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:44:15.3581754Z STAGE:2022-11-23 01:35:06 13836:13836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3582084Z STAGE:2022-11-23 01:35:06 13835:13835 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3582434Z STAGE:2022-11-23 01:35:06 13836:13836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3582778Z STAGE:2022-11-23 01:35:06 13835:13835 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3583105Z STAGE:2022-11-23 01:35:06 13835:13835 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3583440Z STAGE:2022-11-23 01:35:07 13835:13835 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3583787Z STAGE:2022-11-23 01:35:07 13835:13835 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3583889Z ok (6.271s)
2022-11-23T01:44:15.3583910Z 
2022-11-23T01:44:15.3584176Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3584272Z Ran 1 test in 6.272s
2022-11-23T01:44:15.3584291Z 
2022-11-23T01:44:15.3584383Z OK
2022-11-23T01:44:15.3584453Z 
2022-11-23T01:44:15.3584584Z Generating XML reports...
2022-11-23T01:44:15.3585043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013501.xml
2022-11-23T01:44:15.3585413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3585589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3585969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3586166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3586185Z 
2022-11-23T01:44:15.3586293Z Running tests...
2022-11-23T01:44:15.3586539Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3586855Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3587124Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3587346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13957
2022-11-23T01:44:15.3587565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13958
2022-11-23T01:44:15.3587935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3588109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3588527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3588711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3589083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3589255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3589630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3589823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3590069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3590311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3590711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3591108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3591323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3591551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3591813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2euzbfc
2022-11-23T01:44:15.3592083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2euzbfc/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3592335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdq10dom0
2022-11-23T01:44:15.3592603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdq10dom0/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3592705Z ok (5.564s)
2022-11-23T01:44:15.3592725Z 
2022-11-23T01:44:15.3592993Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3593089Z Ran 1 test in 5.564s
2022-11-23T01:44:15.3593124Z 
2022-11-23T01:44:15.3593201Z OK
2022-11-23T01:44:15.3593220Z 
2022-11-23T01:44:15.3593342Z Generating XML reports...
2022-11-23T01:44:15.3593793Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013510.xml
2022-11-23T01:44:15.3594226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3594400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3594779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3594969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3594988Z 
2022-11-23T01:44:15.3595097Z Running tests...
2022-11-23T01:44:15.3595343Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3595656Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3595929Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3596147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14072
2022-11-23T01:44:15.3596370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14073
2022-11-23T01:44:15.3596737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3596912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3597287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3597554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3597934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3598110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3598484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3598678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3598922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3599166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3599568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3599967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3600181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3600409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3600666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhzxqs2w
2022-11-23T01:44:15.3600939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhzxqs2w/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3601191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwolh9y1g
2022-11-23T01:44:15.3601456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwolh9y1g/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3601559Z ok (5.652s)
2022-11-23T01:44:15.3601579Z 
2022-11-23T01:44:15.3601845Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3601961Z Ran 1 test in 5.653s
2022-11-23T01:44:15.3601980Z 
2022-11-23T01:44:15.3602056Z OK
2022-11-23T01:44:15.3602075Z 
2022-11-23T01:44:15.3602197Z Generating XML reports...
2022-11-23T01:44:15.3602649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013518.xml
2022-11-23T01:44:15.3603019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3603388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3603776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3603969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3603989Z 
2022-11-23T01:44:15.3604099Z Running tests...
2022-11-23T01:44:15.3604342Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3604659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3604933Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3605679Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.609s)
2022-11-23T01:44:15.3605702Z 
2022-11-23T01:44:15.3605962Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3606075Z Ran 1 test in 1.609s
2022-11-23T01:44:15.3606094Z 
2022-11-23T01:44:15.3606206Z OK (skipped=1)
2022-11-23T01:44:15.3606225Z 
2022-11-23T01:44:15.3606351Z Generating XML reports...
2022-11-23T01:44:15.3606848Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013526.xml
2022-11-23T01:44:15.3607230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3607389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3607771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3607966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3607985Z 
2022-11-23T01:44:15.3608093Z Running tests...
2022-11-23T01:44:15.3608356Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3608668Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3608925Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3609665Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.619s)
2022-11-23T01:44:15.3609685Z 
2022-11-23T01:44:15.3609949Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3610049Z Ran 1 test in 1.620s
2022-11-23T01:44:15.3610086Z 
2022-11-23T01:44:15.3610177Z OK (skipped=1)
2022-11-23T01:44:15.3610195Z 
2022-11-23T01:44:15.3610318Z Generating XML reports...
2022-11-23T01:44:15.3610765Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013530.xml
2022-11-23T01:44:15.3611133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3611310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3611915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3612114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3612134Z 
2022-11-23T01:44:15.3612245Z Running tests...
2022-11-23T01:44:15.3612492Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3612912Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3613203Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3613943Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T01:44:15.3613964Z 
2022-11-23T01:44:15.3614229Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3614342Z Ran 1 test in 1.642s
2022-11-23T01:44:15.3614361Z 
2022-11-23T01:44:15.3614469Z OK (skipped=1)
2022-11-23T01:44:15.3614488Z 
2022-11-23T01:44:15.3614612Z Generating XML reports...
2022-11-23T01:44:15.3615051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013534.xml
2022-11-23T01:44:15.3615424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3615582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3615962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3616153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3616172Z 
2022-11-23T01:44:15.3616345Z Running tests...
2022-11-23T01:44:15.3616621Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3616934Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3617226Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3617452Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14289
2022-11-23T01:44:15.3617654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14290
2022-11-23T01:44:15.3618028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3618203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3618580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3618775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3619135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3619308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3619679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3619870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3620102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3620346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3620745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3621142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3621373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3621599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3621856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3_zgzpl
2022-11-23T01:44:15.3622189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3_zgzpl/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3622446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8u24wi7
2022-11-23T01:44:15.3622694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8u24wi7/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3622796Z ok (6.269s)
2022-11-23T01:44:15.3622816Z 
2022-11-23T01:44:15.3623087Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3623203Z Ran 1 test in 6.269s
2022-11-23T01:44:15.3623222Z 
2022-11-23T01:44:15.3623314Z OK
2022-11-23T01:44:15.3623333Z 
2022-11-23T01:44:15.3623456Z Generating XML reports...
2022-11-23T01:44:15.3623908Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013538.xml
2022-11-23T01:44:15.3624281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3624444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3624825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3625017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3625036Z 
2022-11-23T01:44:15.3625141Z Running tests...
2022-11-23T01:44:15.3625404Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3625763Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3626046Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3626272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14408
2022-11-23T01:44:15.3626495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14409
2022-11-23T01:44:15.3626855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3627033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3627410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3627600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3627966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3628186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3628564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3628755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3628987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3629231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3629630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3630028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3630261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3630642Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T01:44:15.3630898Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T01:44:15.3631124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3631554Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T01:44:15.3631790Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T01:44:15.3632049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv12alcz
2022-11-23T01:44:15.3632317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv12alcz/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3632575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6rf2izm
2022-11-23T01:44:15.3632844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6rf2izm/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3632949Z ok (5.646s)
2022-11-23T01:44:15.3632969Z 
2022-11-23T01:44:15.3633235Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3633348Z Ran 1 test in 5.646s
2022-11-23T01:44:15.3633371Z 
2022-11-23T01:44:15.3633447Z OK
2022-11-23T01:44:15.3633483Z 
2022-11-23T01:44:15.3633590Z Generating XML reports...
2022-11-23T01:44:15.3634041Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013547.xml
2022-11-23T01:44:15.3634407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3634582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3635005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3635206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3635225Z 
2022-11-23T01:44:15.3635336Z Running tests...
2022-11-23T01:44:15.3635602Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3635900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3636167Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3636391Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14523
2022-11-23T01:44:15.3636608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14524
2022-11-23T01:44:15.3636977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3637157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3637526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3637757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3638122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3638319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3638695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3638886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3639130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3639374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3639775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3640173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3640402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3640686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3640914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3641157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3641557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3641953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3642058Z ok (4.366s)
2022-11-23T01:44:15.3642078Z 
2022-11-23T01:44:15.3642341Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3642452Z Ran 1 test in 4.366s
2022-11-23T01:44:15.3642472Z 
2022-11-23T01:44:15.3642564Z OK
2022-11-23T01:44:15.3642583Z 
2022-11-23T01:44:15.3642693Z Generating XML reports...
2022-11-23T01:44:15.3643143Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013555.xml
2022-11-23T01:44:15.3643512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3643689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3644069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3644306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3644327Z 
2022-11-23T01:44:15.3644444Z Running tests...
2022-11-23T01:44:15.3644711Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3645027Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3645264Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3645491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14630
2022-11-23T01:44:15.3645712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14631
2022-11-23T01:44:15.3646083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3646256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3646636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3646828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3647192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3647348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3647721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3647905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3648149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3648391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3648790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3649185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3649415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3649653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3649918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3650163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3650564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3650956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3651063Z ok (4.351s)
2022-11-23T01:44:15.3651083Z 
2022-11-23T01:44:15.3651346Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3651692Z Ran 1 test in 4.351s
2022-11-23T01:44:15.3651715Z 
2022-11-23T01:44:15.3651814Z OK
2022-11-23T01:44:15.3651833Z 
2022-11-23T01:44:15.3651960Z Generating XML reports...
2022-11-23T01:44:15.3652402Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013602.xml
2022-11-23T01:44:15.3652778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3652953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3653333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3653522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3653542Z 
2022-11-23T01:44:15.3653732Z Running tests...
2022-11-23T01:44:15.3654006Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3654322Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3654583Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3655340Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.606s)
2022-11-23T01:44:15.3655378Z 
2022-11-23T01:44:15.3655620Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3655733Z Ran 1 test in 1.606s
2022-11-23T01:44:15.3655753Z 
2022-11-23T01:44:15.3655862Z OK (skipped=1)
2022-11-23T01:44:15.3655881Z 
2022-11-23T01:44:15.3656009Z Generating XML reports...
2022-11-23T01:44:15.3656455Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013609.xml
2022-11-23T01:44:15.3656823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3656999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3657383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3657555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3657574Z 
2022-11-23T01:44:15.3657681Z Running tests...
2022-11-23T01:44:15.3657942Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3658252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3658530Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3659272Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.603s)
2022-11-23T01:44:15.3659360Z 
2022-11-23T01:44:15.3659633Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3659748Z Ran 1 test in 1.603s
2022-11-23T01:44:15.3659768Z 
2022-11-23T01:44:15.3659877Z OK (skipped=1)
2022-11-23T01:44:15.3659896Z 
2022-11-23T01:44:15.3660020Z Generating XML reports...
2022-11-23T01:44:15.3660454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013613.xml
2022-11-23T01:44:15.3660827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3661003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3661384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3661574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3661598Z 
2022-11-23T01:44:15.3661706Z Running tests...
2022-11-23T01:44:15.3661967Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3662280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3662534Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3662755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14805
2022-11-23T01:44:15.3663018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14806
2022-11-23T01:44:15.3663403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3663580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3663962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3664159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3664520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3664694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3665050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3665241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3665490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3665736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3666135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3666534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3666764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3666990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3667076Z ok (4.252s)
2022-11-23T01:44:15.3667112Z 
2022-11-23T01:44:15.3667360Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3667472Z Ran 1 test in 4.252s
2022-11-23T01:44:15.3667496Z 
2022-11-23T01:44:15.3667585Z OK
2022-11-23T01:44:15.3667604Z 
2022-11-23T01:44:15.3667727Z Generating XML reports...
2022-11-23T01:44:15.3668178Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013618.xml
2022-11-23T01:44:15.3668544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3668777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3669159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3669334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3669353Z 
2022-11-23T01:44:15.3669463Z Running tests...
2022-11-23T01:44:15.3669726Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3670040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3670290Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3670310Z 
2022-11-23T01:44:15.3670569Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3670679Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3670702Z 
2022-11-23T01:44:15.3670808Z OK (skipped=1)
2022-11-23T01:44:15.3670826Z 
2022-11-23T01:44:15.3670948Z Generating XML reports...
2022-11-23T01:44:15.3671382Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013624.xml
2022-11-23T01:44:15.3671753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3671928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3672351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3672550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3672570Z 
2022-11-23T01:44:15.3672680Z Running tests...
2022-11-23T01:44:15.3672946Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3673261Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3673510Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3673548Z 
2022-11-23T01:44:15.3673790Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3673900Z Ran 1 test in 0.003s
2022-11-23T01:44:15.3673920Z 
2022-11-23T01:44:15.3674025Z OK (skipped=1)
2022-11-23T01:44:15.3674044Z 
2022-11-23T01:44:15.3674165Z Generating XML reports...
2022-11-23T01:44:15.3674618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013627.xml
2022-11-23T01:44:15.3674987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3675162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3675539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3675717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3675736Z 
2022-11-23T01:44:15.3675844Z Running tests...
2022-11-23T01:44:15.3676106Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3676418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3676665Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3676888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14976
2022-11-23T01:44:15.3677106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14977
2022-11-23T01:44:15.3677476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3677635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3678082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3678277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3678640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3678814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3679190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3679384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3679633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3679879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3680264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3680660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3680888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3681115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3681493Z STAGE:2022-11-23 01:36:34 14977:14977 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3681829Z STAGE:2022-11-23 01:36:34 14976:14976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3682166Z STAGE:2022-11-23 01:36:35 14976:14976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3682518Z STAGE:2022-11-23 01:36:35 14976:14976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3682856Z STAGE:2022-11-23 01:36:35 14977:14977 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3683183Z STAGE:2022-11-23 01:36:35 14977:14977 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3683514Z STAGE:2022-11-23 01:36:35 14976:14976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3683830Z STAGE:2022-11-23 01:36:35 14977:14977 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3684168Z STAGE:2022-11-23 01:36:35 14977:14977 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3684511Z STAGE:2022-11-23 01:36:35 14977:14977 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3684840Z STAGE:2022-11-23 01:36:35 14976:14976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3685184Z STAGE:2022-11-23 01:36:35 14976:14976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3685291Z ok (6.524s)
2022-11-23T01:44:15.3685311Z 
2022-11-23T01:44:15.3685576Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3685672Z Ran 1 test in 6.524s
2022-11-23T01:44:15.3685691Z 
2022-11-23T01:44:15.3685783Z OK
2022-11-23T01:44:15.3685802Z 
2022-11-23T01:44:15.3685925Z Generating XML reports...
2022-11-23T01:44:15.3686376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013629.xml
2022-11-23T01:44:15.3686750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3686926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3687306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3687497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3687567Z 
2022-11-23T01:44:15.3687664Z Running tests...
2022-11-23T01:44:15.3687930Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3688244Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3688510Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3688530Z 
2022-11-23T01:44:15.3688788Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3688904Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3688924Z 
2022-11-23T01:44:15.3689030Z OK (skipped=1)
2022-11-23T01:44:15.3689049Z 
2022-11-23T01:44:15.3689171Z Generating XML reports...
2022-11-23T01:44:15.3689616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013638.xml
2022-11-23T01:44:15.3689970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3690150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3690529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3690719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3690738Z 
2022-11-23T01:44:15.3690846Z Running tests...
2022-11-23T01:44:15.3691109Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3691723Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3692018Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3692039Z 
2022-11-23T01:44:15.3692313Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3692413Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3692432Z 
2022-11-23T01:44:15.3692541Z OK (skipped=1)
2022-11-23T01:44:15.3692560Z 
2022-11-23T01:44:15.3692684Z Generating XML reports...
2022-11-23T01:44:15.3693130Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013641.xml
2022-11-23T01:44:15.3693496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3693735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3694160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3694389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3694409Z 
2022-11-23T01:44:15.3694532Z Running tests...
2022-11-23T01:44:15.3694780Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3695240Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3695528Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3695840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15165
2022-11-23T01:44:15.3696096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15166
2022-11-23T01:44:15.3696503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3696719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3697138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3697313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3697715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3698056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3698479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3698703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3698989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3699272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3699713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3700145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3700362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3700641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3700815Z ok (5.729s)
2022-11-23T01:44:15.3700837Z 
2022-11-23T01:44:15.3701180Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3701327Z Ran 1 test in 5.730s
2022-11-23T01:44:15.3701347Z 
2022-11-23T01:44:15.3701473Z OK
2022-11-23T01:44:15.3701494Z 
2022-11-23T01:44:15.3701652Z Generating XML reports...
2022-11-23T01:44:15.3702188Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013643.xml
2022-11-23T01:44:15.3702623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3702787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3703201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3703469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3703489Z 
2022-11-23T01:44:15.3703634Z Running tests...
2022-11-23T01:44:15.3703931Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3704279Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3704585Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3705385Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.624s)
2022-11-23T01:44:15.3705406Z 
2022-11-23T01:44:15.3705704Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3705806Z Ran 1 test in 1.625s
2022-11-23T01:44:15.3705826Z 
2022-11-23T01:44:15.3705970Z OK (skipped=1)
2022-11-23T01:44:15.3705989Z 
2022-11-23T01:44:15.3706215Z Generating XML reports...
2022-11-23T01:44:15.3706703Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013651.xml
2022-11-23T01:44:15.3707112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3707326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3707752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3707980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3708000Z 
2022-11-23T01:44:15.3708142Z Running tests...
2022-11-23T01:44:15.3708392Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3708803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3709125Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3709381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15317
2022-11-23T01:44:15.3709644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15318
2022-11-23T01:44:15.3710056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3710266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3710679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3710855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3711257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3711500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3711938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3712174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3712457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3712787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3713237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3713673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3713893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3714170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3714426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3714742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3715179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3715607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3715746Z ok (4.427s)
2022-11-23T01:44:15.3715766Z 
2022-11-23T01:44:15.3716066Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3716216Z Ran 1 test in 4.428s
2022-11-23T01:44:15.3716239Z 
2022-11-23T01:44:15.3716317Z OK
2022-11-23T01:44:15.3716336Z 
2022-11-23T01:44:15.3716527Z Generating XML reports...
2022-11-23T01:44:15.3717025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013655.xml
2022-11-23T01:44:15.3717463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3717674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3718094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3728462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3728491Z 
2022-11-23T01:44:15.3728640Z Running tests...
2022-11-23T01:44:15.3728947Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3729274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3729638Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3729845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15424
2022-11-23T01:44:15.3730065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15425
2022-11-23T01:44:15.3730450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3730634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3731019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3731214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3731894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3732085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3732474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3732649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3732896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3733143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3733640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3734058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3734290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3734529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3734633Z ok (5.623s)
2022-11-23T01:44:15.3734654Z 
2022-11-23T01:44:15.3734905Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3735017Z Ran 1 test in 5.623s
2022-11-23T01:44:15.3735036Z 
2022-11-23T01:44:15.3735130Z OK
2022-11-23T01:44:15.3735150Z 
2022-11-23T01:44:15.3735274Z Generating XML reports...
2022-11-23T01:44:15.3735733Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013702.xml
2022-11-23T01:44:15.3736107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3736285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3736663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3736858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3736877Z 
2022-11-23T01:44:15.3736968Z Running tests...
2022-11-23T01:44:15.3737233Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3737549Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3737793Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3738016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15539
2022-11-23T01:44:15.3738231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15540
2022-11-23T01:44:15.3738602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3738776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3739216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3739407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3739771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3739948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3740317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3740511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3740759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3741006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3741406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3741791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3742022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3742247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3742351Z ok (4.421s)
2022-11-23T01:44:15.3742370Z 
2022-11-23T01:44:15.3742679Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3742801Z Ran 1 test in 4.421s
2022-11-23T01:44:15.3742822Z 
2022-11-23T01:44:15.3742915Z OK
2022-11-23T01:44:15.3742934Z 
2022-11-23T01:44:15.3743057Z Generating XML reports...
2022-11-23T01:44:15.3743495Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013711.xml
2022-11-23T01:44:15.3743873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3744050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3744430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3744622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3744641Z 
2022-11-23T01:44:15.3744750Z Running tests...
2022-11-23T01:44:15.3745018Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3745334Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3745602Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3745806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15644
2022-11-23T01:44:15.3746030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15645
2022-11-23T01:44:15.3746398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3746572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3746949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3747139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3747503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3747676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3748030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3748278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3748525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3748771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3749172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3749568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3749797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3750035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3750261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3750490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3750888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3751279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3751385Z ok (4.338s)
2022-11-23T01:44:15.3751404Z 
2022-11-23T01:44:15.3751670Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3751827Z Ran 1 test in 4.338s
2022-11-23T01:44:15.3751848Z 
2022-11-23T01:44:15.3751943Z OK
2022-11-23T01:44:15.3751962Z 
2022-11-23T01:44:15.3752087Z Generating XML reports...
2022-11-23T01:44:15.3752542Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013718.xml
2022-11-23T01:44:15.3752899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3753080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3753462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3753655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3753674Z 
2022-11-23T01:44:15.3753783Z Running tests...
2022-11-23T01:44:15.3754043Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3754357Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3754618Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3754837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15751
2022-11-23T01:44:15.3755040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15752
2022-11-23T01:44:15.3755420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3755595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3755972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3756164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3756530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3756703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3757074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3757247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3757567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3757811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3758216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3758612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3758846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3759092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3759316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3759554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3759937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3760332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3760435Z ok (4.307s)
2022-11-23T01:44:15.3760455Z 
2022-11-23T01:44:15.3760722Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3760834Z Ran 1 test in 4.307s
2022-11-23T01:44:15.3760854Z 
2022-11-23T01:44:15.3760945Z OK
2022-11-23T01:44:15.3761013Z 
2022-11-23T01:44:15.3761144Z Generating XML reports...
2022-11-23T01:44:15.3761598Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013724.xml
2022-11-23T01:44:15.3761969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3762129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3762508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3762700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3762720Z 
2022-11-23T01:44:15.3762829Z Running tests...
2022-11-23T01:44:15.3763091Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3763405Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3763671Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3763892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15858
2022-11-23T01:44:15.3764095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15859
2022-11-23T01:44:15.3764467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3764642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3765020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3765210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3765575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3765749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3766121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3766312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3766541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3766839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3767243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3767639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3767870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3768103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3768362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_gh0cig7
2022-11-23T01:44:15.3768631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_gh0cig7/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3768886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cutdh3p
2022-11-23T01:44:15.3769142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cutdh3p/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3769244Z ok (5.602s)
2022-11-23T01:44:15.3769264Z 
2022-11-23T01:44:15.3769533Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3769646Z Ran 1 test in 5.602s
2022-11-23T01:44:15.3769666Z 
2022-11-23T01:44:15.3769758Z OK
2022-11-23T01:44:15.3769777Z 
2022-11-23T01:44:15.3769900Z Generating XML reports...
2022-11-23T01:44:15.3770397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013731.xml
2022-11-23T01:44:15.3770777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3770937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3771316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3771753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3771776Z 
2022-11-23T01:44:15.3771893Z Running tests...
2022-11-23T01:44:15.3772161Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3772476Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3772715Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-11-23T01:44:15.3772735Z 
2022-11-23T01:44:15.3772997Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3773109Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3773128Z 
2022-11-23T01:44:15.3773219Z OK (skipped=1)
2022-11-23T01:44:15.3773238Z 
2022-11-23T01:44:15.3773362Z Generating XML reports...
2022-11-23T01:44:15.3773816Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013739.xml
2022-11-23T01:44:15.3774191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3774365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3774744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3774936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3774957Z 
2022-11-23T01:44:15.3775065Z Running tests...
2022-11-23T01:44:15.3775314Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3775631Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3775868Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T01:44:15.3775888Z 
2022-11-23T01:44:15.3776239Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3776351Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3776371Z 
2022-11-23T01:44:15.3776481Z OK (skipped=1)
2022-11-23T01:44:15.3776499Z 
2022-11-23T01:44:15.3776622Z Generating XML reports...
2022-11-23T01:44:15.3777071Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013742.xml
2022-11-23T01:44:15.3777440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3777602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3777982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3778174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3778193Z 
2022-11-23T01:44:15.3778300Z Running tests...
2022-11-23T01:44:15.3778564Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3778878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3779144Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T01:44:15.3779164Z 
2022-11-23T01:44:15.3779422Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3779534Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3779553Z 
2022-11-23T01:44:15.3779644Z OK (skipped=1)
2022-11-23T01:44:15.3779662Z 
2022-11-23T01:44:15.3779848Z Generating XML reports...
2022-11-23T01:44:15.3780315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013744.xml
2022-11-23T01:44:15.3780683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3780858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3781244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3781435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3781455Z 
2022-11-23T01:44:15.3781562Z Running tests...
2022-11-23T01:44:15.3781812Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3782126Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3782386Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T01:44:15.3782406Z 
2022-11-23T01:44:15.3782663Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3782775Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3782794Z 
2022-11-23T01:44:15.3782901Z OK (skipped=1)
2022-11-23T01:44:15.3782923Z 
2022-11-23T01:44:15.3783050Z Generating XML reports...
2022-11-23T01:44:15.3783498Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013747.xml
2022-11-23T01:44:15.3783866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3784024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3784402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3784595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3784615Z 
2022-11-23T01:44:15.3784722Z Running tests...
2022-11-23T01:44:15.3784986Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3785300Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3785641Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3785862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16108
2022-11-23T01:44:15.3786081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16109
2022-11-23T01:44:15.3786443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3786619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3787003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3787195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3787556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3787730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3788109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3788298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3788530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3788774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3789218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3789625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3789858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3790101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3790328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3790565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3790962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3791336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3791580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:44:15.3791818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:44:15.3792204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3792600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3792833Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:44:15.3792936Z ok (21.686s)
2022-11-23T01:44:15.3792956Z 
2022-11-23T01:44:15.3793222Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3793334Z Ran 1 test in 21.686s
2022-11-23T01:44:15.3793354Z 
2022-11-23T01:44:15.3793430Z OK
2022-11-23T01:44:15.3793448Z 
2022-11-23T01:44:15.3793577Z Generating XML reports...
2022-11-23T01:44:15.3794031Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013749.xml
2022-11-23T01:44:15.3794403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3794580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3795023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3795216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3795235Z 
2022-11-23T01:44:15.3795347Z Running tests...
2022-11-23T01:44:15.3795607Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3795904Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3796207Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3796430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16231
2022-11-23T01:44:15.3796650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16232
2022-11-23T01:44:15.3797016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3797193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3797569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3797760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3798108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3798328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3798711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3798899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3799145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3799390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3799787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3800184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3800413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3800628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3800867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.3801109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.3801507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3801901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.3802140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:44:15.3802378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:44:15.3802772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3803166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.3803382Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:44:15.3803612Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:44:15.3803768Z ok (21.384s)
2022-11-23T01:44:15.3803788Z 
2022-11-23T01:44:15.3804056Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3804170Z Ran 1 test in 21.384s
2022-11-23T01:44:15.3804189Z 
2022-11-23T01:44:15.3804283Z OK
2022-11-23T01:44:15.3804302Z 
2022-11-23T01:44:15.3804427Z Generating XML reports...
2022-11-23T01:44:15.3804879Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013813.xml
2022-11-23T01:44:15.3805252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3805412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3805791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3805982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3806005Z 
2022-11-23T01:44:15.3806113Z Running tests...
2022-11-23T01:44:15.3806375Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3806688Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3807097Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.3807117Z 
2022-11-23T01:44:15.3807377Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3807472Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3807554Z 
2022-11-23T01:44:15.3807652Z OK (skipped=1)
2022-11-23T01:44:15.3807671Z 
2022-11-23T01:44:15.3807795Z Generating XML reports...
2022-11-23T01:44:15.3808246Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013837.xml
2022-11-23T01:44:15.3808614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3808795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3809174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3809367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3809386Z 
2022-11-23T01:44:15.3809495Z Running tests...
2022-11-23T01:44:15.3809741Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3810056Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3810444Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.3810464Z 
2022-11-23T01:44:15.3810718Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3810831Z Ran 1 test in 0.003s
2022-11-23T01:44:15.3810850Z 
2022-11-23T01:44:15.3810958Z OK (skipped=1)
2022-11-23T01:44:15.3810977Z 
2022-11-23T01:44:15.3811101Z Generating XML reports...
2022-11-23T01:44:15.3811772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013840.xml
2022-11-23T01:44:15.3812157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3812318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3812703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3812896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3812916Z 
2022-11-23T01:44:15.3813025Z Running tests...
2022-11-23T01:44:15.3813286Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3813691Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3814110Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.3814130Z 
2022-11-23T01:44:15.3814389Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3814501Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3814521Z 
2022-11-23T01:44:15.3814610Z OK (skipped=1)
2022-11-23T01:44:15.3814629Z 
2022-11-23T01:44:15.3814753Z Generating XML reports...
2022-11-23T01:44:15.3815204Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013842.xml
2022-11-23T01:44:15.3815572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3815747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3816126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3816317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3816336Z 
2022-11-23T01:44:15.3816444Z Running tests...
2022-11-23T01:44:15.3816690Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3817005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3817474Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.3817497Z 
2022-11-23T01:44:15.3817768Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3817881Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3817900Z 
2022-11-23T01:44:15.3818007Z OK (skipped=1)
2022-11-23T01:44:15.3818026Z 
2022-11-23T01:44:15.3818148Z Generating XML reports...
2022-11-23T01:44:15.3818600Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013844.xml
2022-11-23T01:44:15.3818972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3819130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3819510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3819705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3819724Z 
2022-11-23T01:44:15.3819832Z Running tests...
2022-11-23T01:44:15.3820095Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3820408Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3820812Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.3820836Z 
2022-11-23T01:44:15.3821092Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3821204Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3821223Z 
2022-11-23T01:44:15.3821311Z OK (skipped=1)
2022-11-23T01:44:15.3821348Z 
2022-11-23T01:44:15.3821454Z Generating XML reports...
2022-11-23T01:44:15.3821900Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013847.xml
2022-11-23T01:44:15.3822273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3822448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3822823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3823014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3823097Z 
2022-11-23T01:44:15.3823207Z Running tests...
2022-11-23T01:44:15.3823470Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3823765Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3824040Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3824261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16519
2022-11-23T01:44:15.3824484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16520
2022-11-23T01:44:15.3824854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3825030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3825408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3825602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3825948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3826125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3826499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3826765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3827019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3827265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3827669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3828066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3828347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3828561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3828664Z ok (5.119s)
2022-11-23T01:44:15.3828683Z 
2022-11-23T01:44:15.3828948Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3829069Z Ran 1 test in 5.120s
2022-11-23T01:44:15.3829089Z 
2022-11-23T01:44:15.3829181Z OK
2022-11-23T01:44:15.3829200Z 
2022-11-23T01:44:15.3829324Z Generating XML reports...
2022-11-23T01:44:15.3829773Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013849.xml
2022-11-23T01:44:15.3830142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3830320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3830681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3830873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3830892Z 
2022-11-23T01:44:15.3831000Z Running tests...
2022-11-23T01:44:15.3831258Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3831573Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3831846Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3832068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16633
2022-11-23T01:44:15.3832284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16634
2022-11-23T01:44:15.3832698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3832874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3833251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3833445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3833815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3833988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3834361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3834552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3834786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3835031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3835436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3835834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3836110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3836346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3836450Z ok (5.240s)
2022-11-23T01:44:15.3836469Z 
2022-11-23T01:44:15.3836736Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3836851Z Ran 1 test in 5.240s
2022-11-23T01:44:15.3836871Z 
2022-11-23T01:44:15.3836946Z OK
2022-11-23T01:44:15.3836983Z 
2022-11-23T01:44:15.3837090Z Generating XML reports...
2022-11-23T01:44:15.3837541Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013857.xml
2022-11-23T01:44:15.3837912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3838087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3838467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3838658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3838678Z 
2022-11-23T01:44:15.3838788Z Running tests...
2022-11-23T01:44:15.3839046Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3839347Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3839621Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3839840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16747
2022-11-23T01:44:15.3840058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16748
2022-11-23T01:44:15.3840424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3840604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3840981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3841173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3841521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3841744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3842118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3842308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3842553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3842802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3843201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3843594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3843826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3844040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3844144Z ok (5.521s)
2022-11-23T01:44:15.3844164Z 
2022-11-23T01:44:15.3844427Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3844541Z Ran 1 test in 5.521s
2022-11-23T01:44:15.3844561Z 
2022-11-23T01:44:15.3844653Z OK
2022-11-23T01:44:15.3844672Z 
2022-11-23T01:44:15.3844796Z Generating XML reports...
2022-11-23T01:44:15.3845283Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013904.xml
2022-11-23T01:44:15.3845659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3845838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3846202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3846395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3846414Z 
2022-11-23T01:44:15.3846523Z Running tests...
2022-11-23T01:44:15.3846780Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3847093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3847360Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3847585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16862
2022-11-23T01:44:15.3847802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16863
2022-11-23T01:44:15.3848155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3848331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3848710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3848899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3849259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3849431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3849802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3849994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3850222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3850466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3850929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3851325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3851786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3852022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3852124Z ok (5.130s)
2022-11-23T01:44:15.3852145Z 
2022-11-23T01:44:15.3852418Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3852530Z Ran 1 test in 5.130s
2022-11-23T01:44:15.3852550Z 
2022-11-23T01:44:15.3852625Z OK
2022-11-23T01:44:15.3852662Z 
2022-11-23T01:44:15.3852770Z Generating XML reports...
2022-11-23T01:44:15.3853219Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013913.xml
2022-11-23T01:44:15.3853596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3853773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3854155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3854346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3854365Z 
2022-11-23T01:44:15.3854475Z Running tests...
2022-11-23T01:44:15.3854813Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3855126Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3855394Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3856149Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.594s)
2022-11-23T01:44:15.3856174Z 
2022-11-23T01:44:15.3856436Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3856549Z Ran 1 test in 1.594s
2022-11-23T01:44:15.3856568Z 
2022-11-23T01:44:15.3856674Z OK (skipped=1)
2022-11-23T01:44:15.3856693Z 
2022-11-23T01:44:15.3856816Z Generating XML reports...
2022-11-23T01:44:15.3857267Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013920.xml
2022-11-23T01:44:15.3857640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3857799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3858182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3858374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3858393Z 
2022-11-23T01:44:15.3858501Z Running tests...
2022-11-23T01:44:15.3858761Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3859074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3859326Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:44:15.3859346Z 
2022-11-23T01:44:15.3859606Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3859719Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3859738Z 
2022-11-23T01:44:15.3859830Z OK (skipped=1)
2022-11-23T01:44:15.3859864Z 
2022-11-23T01:44:15.3859971Z Generating XML reports...
2022-11-23T01:44:15.3860499Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013924.xml
2022-11-23T01:44:15.3860865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3861040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3861414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3861608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3861629Z 
2022-11-23T01:44:15.3861735Z Running tests...
2022-11-23T01:44:15.3861997Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3862293Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3862563Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:44:15.3862586Z 
2022-11-23T01:44:15.3862848Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3862959Z Ran 1 test in 0.003s
2022-11-23T01:44:15.3862978Z 
2022-11-23T01:44:15.3863085Z OK (skipped=1)
2022-11-23T01:44:15.3863104Z 
2022-11-23T01:44:15.3863225Z Generating XML reports...
2022-11-23T01:44:15.3863669Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013927.xml
2022-11-23T01:44:15.3864085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3864267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3864635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3864827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3864850Z 
2022-11-23T01:44:15.3864958Z Running tests...
2022-11-23T01:44:15.3865218Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3865530Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3865841Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:44:15.3865861Z 
2022-11-23T01:44:15.3866122Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3866236Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3866255Z 
2022-11-23T01:44:15.3866345Z OK (skipped=1)
2022-11-23T01:44:15.3866382Z 
2022-11-23T01:44:15.3866488Z Generating XML reports...
2022-11-23T01:44:15.3866933Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013929.xml
2022-11-23T01:44:15.3867304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3867478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3867853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3868043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3868063Z 
2022-11-23T01:44:15.3868169Z Running tests...
2022-11-23T01:44:15.3868428Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3868729Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3869033Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3869254Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17109
2022-11-23T01:44:15.3869528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17110
2022-11-23T01:44:15.3869902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3870077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3870457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3870649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3871014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3871172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3871545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3871735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3871982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3872230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3872629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3873024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3873299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3873519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3873622Z ok (4.261s)
2022-11-23T01:44:15.3873642Z 
2022-11-23T01:44:15.3873909Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3874026Z Ran 1 test in 4.262s
2022-11-23T01:44:15.3874045Z 
2022-11-23T01:44:15.3874138Z OK
2022-11-23T01:44:15.3874157Z 
2022-11-23T01:44:15.3874281Z Generating XML reports...
2022-11-23T01:44:15.3874727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013932.xml
2022-11-23T01:44:15.3875096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3875272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3875636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3875829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3875848Z 
2022-11-23T01:44:15.3875958Z Running tests...
2022-11-23T01:44:15.3876218Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3876532Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3876825Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3877043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17214
2022-11-23T01:44:15.3877260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17215
2022-11-23T01:44:15.3877619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3877794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3878175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3878367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3878786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3878960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3879333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3879523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3879768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3880000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3880403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3880799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3881032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3881255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3881355Z ok (4.235s)
2022-11-23T01:44:15.3881375Z 
2022-11-23T01:44:15.3881639Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3881751Z Ran 1 test in 4.235s
2022-11-23T01:44:15.3881770Z 
2022-11-23T01:44:15.3881865Z OK
2022-11-23T01:44:15.3881884Z 
2022-11-23T01:44:15.3881992Z Generating XML reports...
2022-11-23T01:44:15.3882488Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013938.xml
2022-11-23T01:44:15.3882870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3883049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3883432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3883623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3883643Z 
2022-11-23T01:44:15.3883751Z Running tests...
2022-11-23T01:44:15.3884016Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3884314Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3884594Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:44:15.3884613Z 
2022-11-23T01:44:15.3884872Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3884983Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3885002Z 
2022-11-23T01:44:15.3885107Z OK (skipped=1)
2022-11-23T01:44:15.3885126Z 
2022-11-23T01:44:15.3885247Z Generating XML reports...
2022-11-23T01:44:15.3885697Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013945.xml
2022-11-23T01:44:15.3886061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3886237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3886599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3886791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3886815Z 
2022-11-23T01:44:15.3886923Z Running tests...
2022-11-23T01:44:15.3887184Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3887499Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3887799Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:44:15.3887879Z 
2022-11-23T01:44:15.3888147Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3888260Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3888279Z 
2022-11-23T01:44:15.3888389Z OK (skipped=1)
2022-11-23T01:44:15.3888408Z 
2022-11-23T01:44:15.3888515Z Generating XML reports...
2022-11-23T01:44:15.3888961Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013948.xml
2022-11-23T01:44:15.3889332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3889507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3889886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3890075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3890098Z 
2022-11-23T01:44:15.3890205Z Running tests...
2022-11-23T01:44:15.3890473Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3890768Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3891045Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3892099Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.672s)
2022-11-23T01:44:15.3892125Z 
2022-11-23T01:44:15.3892408Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3892520Z Ran 1 test in 1.672s
2022-11-23T01:44:15.3892543Z 
2022-11-23T01:44:15.3892651Z OK (skipped=1)
2022-11-23T01:44:15.3892669Z 
2022-11-23T01:44:15.3892793Z Generating XML reports...
2022-11-23T01:44:15.3893239Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013950.xml
2022-11-23T01:44:15.3893607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3893783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3894148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3894340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3894359Z 
2022-11-23T01:44:15.3894468Z Running tests...
2022-11-23T01:44:15.3894728Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3895040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3895322Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3895541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17419
2022-11-23T01:44:15.3895759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17420
2022-11-23T01:44:15.3896131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3896293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3896673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3896862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3897229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3897474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3897851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3898042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3898286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3898513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3898917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3899312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3899545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3899774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3900033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzfwgj88
2022-11-23T01:44:15.3900302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzfwgj88/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3900555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgl9pg5b
2022-11-23T01:44:15.3900871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgl9pg5b/_remote_module_non_scriptable.py
2022-11-23T01:44:15.3900963Z ok (6.169s)
2022-11-23T01:44:15.3900982Z 
2022-11-23T01:44:15.3901252Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3901363Z Ran 1 test in 6.169s
2022-11-23T01:44:15.3901382Z 
2022-11-23T01:44:15.3901475Z OK
2022-11-23T01:44:15.3901494Z 
2022-11-23T01:44:15.3901617Z Generating XML reports...
2022-11-23T01:44:15.3902068Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013954.xml
2022-11-23T01:44:15.3902436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3902614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3902992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3903169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3903188Z 
2022-11-23T01:44:15.3903298Z Running tests...
2022-11-23T01:44:15.3903561Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3903872Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3904136Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3904359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17538
2022-11-23T01:44:15.3904574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17539
2022-11-23T01:44:15.3904939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3905098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3905479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3905672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3906030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3906203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3906639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3906826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3907072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3907318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3907702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3908097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3908329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3908559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3908663Z ok (5.771s)
2022-11-23T01:44:15.3908683Z 
2022-11-23T01:44:15.3908951Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3909063Z Ran 1 test in 5.771s
2022-11-23T01:44:15.3909083Z 
2022-11-23T01:44:15.3909175Z OK
2022-11-23T01:44:15.3909196Z 
2022-11-23T01:44:15.3909302Z Generating XML reports...
2022-11-23T01:44:15.3909749Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014003.xml
2022-11-23T01:44:15.3910164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3910346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3910725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3910918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3910940Z 
2022-11-23T01:44:15.3911049Z Running tests...
2022-11-23T01:44:15.3911310Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3911622Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3911892Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3912112Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17653
2022-11-23T01:44:15.3912334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17654
2022-11-23T01:44:15.3912706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3912879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3913256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3913449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3913813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3913971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3914345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3914533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3914782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3915025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3915423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3915877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3916109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3916337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3916422Z ok (5.616s)
2022-11-23T01:44:15.3916441Z 
2022-11-23T01:44:15.3916706Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3916819Z Ran 1 test in 5.616s
2022-11-23T01:44:15.3916841Z 
2022-11-23T01:44:15.3916933Z OK
2022-11-23T01:44:15.3916953Z 
2022-11-23T01:44:15.3917077Z Generating XML reports...
2022-11-23T01:44:15.3917525Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014011.xml
2022-11-23T01:44:15.3917893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3918073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3918452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3918627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3918645Z 
2022-11-23T01:44:15.3918755Z Running tests...
2022-11-23T01:44:15.3919020Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3919375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3919660Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3920406Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s)
2022-11-23T01:44:15.3920430Z 
2022-11-23T01:44:15.3920688Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3920800Z Ran 1 test in 1.614s
2022-11-23T01:44:15.3920819Z 
2022-11-23T01:44:15.3920925Z OK (skipped=1)
2022-11-23T01:44:15.3920944Z 
2022-11-23T01:44:15.3921051Z Generating XML reports...
2022-11-23T01:44:15.3921498Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014019.xml
2022-11-23T01:44:15.3921869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3922042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3922421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3922613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3922633Z 
2022-11-23T01:44:15.3922740Z Running tests...
2022-11-23T01:44:15.3923002Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3923316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3923595Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3924340Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2022-11-23T01:44:15.3924360Z 
2022-11-23T01:44:15.3924620Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3924787Z Ran 1 test in 1.627s
2022-11-23T01:44:15.3924806Z 
2022-11-23T01:44:15.3924913Z OK (skipped=1)
2022-11-23T01:44:15.3924932Z 
2022-11-23T01:44:15.3925053Z Generating XML reports...
2022-11-23T01:44:15.3925503Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014023.xml
2022-11-23T01:44:15.3925874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3926051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3926432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3926606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3926625Z 
2022-11-23T01:44:15.3926732Z Running tests...
2022-11-23T01:44:15.3926997Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3927313Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3927622Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3927841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17836
2022-11-23T01:44:15.3928057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17837
2022-11-23T01:44:15.3928526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3928687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3929073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3929263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3929623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3929798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3930169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3930355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3930600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3930847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3931232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3931861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3932106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3932335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3932484Z skip: Need at least 4 CUDA devices (4.254s)
2022-11-23T01:44:15.3932504Z 
2022-11-23T01:44:15.3932772Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3932887Z Ran 1 test in 4.255s
2022-11-23T01:44:15.3932906Z 
2022-11-23T01:44:15.3933015Z OK (skipped=1)
2022-11-23T01:44:15.3933034Z 
2022-11-23T01:44:15.3933161Z Generating XML reports...
2022-11-23T01:44:15.3933595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014028.xml
2022-11-23T01:44:15.3933964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3934140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3934606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3934796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3934816Z 
2022-11-23T01:44:15.3934925Z Running tests...
2022-11-23T01:44:15.3935190Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3935507Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3935822Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3936042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17941
2022-11-23T01:44:15.3936257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17942
2022-11-23T01:44:15.3936626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3936803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3937180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3937370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3937782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3938023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3938402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3938592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3938838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3939087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3939484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3939879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3940109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3940340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3940491Z skip: Need at least 4 CUDA devices (4.366s)
2022-11-23T01:44:15.3940511Z 
2022-11-23T01:44:15.3940758Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3940869Z Ran 1 test in 4.367s
2022-11-23T01:44:15.3940889Z 
2022-11-23T01:44:15.3940996Z OK (skipped=1)
2022-11-23T01:44:15.3941018Z 
2022-11-23T01:44:15.3941140Z Generating XML reports...
2022-11-23T01:44:15.3941591Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014034.xml
2022-11-23T01:44:15.3941956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3942129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3942506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3942685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3942720Z 
2022-11-23T01:44:15.3942812Z Running tests...
2022-11-23T01:44:15.3943075Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3943385Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3943733Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3944481Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s)
2022-11-23T01:44:15.3944501Z 
2022-11-23T01:44:15.3944762Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3944879Z Ran 1 test in 1.614s
2022-11-23T01:44:15.3944898Z 
2022-11-23T01:44:15.3945007Z OK (skipped=1)
2022-11-23T01:44:15.3945026Z 
2022-11-23T01:44:15.3945149Z Generating XML reports...
2022-11-23T01:44:15.3945580Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014041.xml
2022-11-23T01:44:15.3945948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3946129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3946503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3946694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3946714Z 
2022-11-23T01:44:15.3946823Z Running tests...
2022-11-23T01:44:15.3947082Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3947441Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3947701Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3947738Z 
2022-11-23T01:44:15.3947984Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3948101Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3948121Z 
2022-11-23T01:44:15.3948227Z OK (skipped=1)
2022-11-23T01:44:15.3948246Z 
2022-11-23T01:44:15.3948368Z Generating XML reports...
2022-11-23T01:44:15.3948813Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014045.xml
2022-11-23T01:44:15.3949180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3949353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3949731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3949905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3949940Z 
2022-11-23T01:44:15.3950031Z Running tests...
2022-11-23T01:44:15.3950291Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3950605Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3950871Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3950890Z 
2022-11-23T01:44:15.3951147Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3951255Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3951274Z 
2022-11-23T01:44:15.3951381Z OK (skipped=1)
2022-11-23T01:44:15.3951400Z 
2022-11-23T01:44:15.3951522Z Generating XML reports...
2022-11-23T01:44:15.3951949Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014048.xml
2022-11-23T01:44:15.3952316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3952490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3952935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3953126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3953145Z 
2022-11-23T01:44:15.3953253Z Running tests...
2022-11-23T01:44:15.3953511Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3953820Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3954100Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3954119Z 
2022-11-23T01:44:15.3954361Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3954471Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3954490Z 
2022-11-23T01:44:15.3954599Z OK (skipped=1)
2022-11-23T01:44:15.3954617Z 
2022-11-23T01:44:15.3954740Z Generating XML reports...
2022-11-23T01:44:15.3955189Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014050.xml
2022-11-23T01:44:15.3955559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3955734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3956112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3956304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3956370Z 
2022-11-23T01:44:15.3956465Z Running tests...
2022-11-23T01:44:15.3956729Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3957042Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3957309Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3957332Z 
2022-11-23T01:44:15.3957591Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3957703Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3957722Z 
2022-11-23T01:44:15.3957828Z OK (skipped=1)
2022-11-23T01:44:15.3957847Z 
2022-11-23T01:44:15.3957969Z Generating XML reports...
2022-11-23T01:44:15.3958399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014053.xml
2022-11-23T01:44:15.3958774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3958951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3959331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3959524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3959546Z 
2022-11-23T01:44:15.3959654Z Running tests...
2022-11-23T01:44:15.3959912Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3960224Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3960489Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3960508Z 
2022-11-23T01:44:15.3960747Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3960856Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3960878Z 
2022-11-23T01:44:15.3960985Z OK (skipped=1)
2022-11-23T01:44:15.3961004Z 
2022-11-23T01:44:15.3961128Z Generating XML reports...
2022-11-23T01:44:15.3961572Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014055.xml
2022-11-23T01:44:15.3961937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3962164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3962548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3962739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3962759Z 
2022-11-23T01:44:15.3962849Z Running tests...
2022-11-23T01:44:15.3963113Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3963426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3963690Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3963709Z 
2022-11-23T01:44:15.3963966Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3964077Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3964100Z 
2022-11-23T01:44:15.3964205Z OK (skipped=1)
2022-11-23T01:44:15.3964224Z 
2022-11-23T01:44:15.3964348Z Generating XML reports...
2022-11-23T01:44:15.3964777Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014057.xml
2022-11-23T01:44:15.3965148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3965323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3965745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3965941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3965960Z 
2022-11-23T01:44:15.3966068Z Running tests...
2022-11-23T01:44:15.3966330Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3966641Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3966912Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3966931Z 
2022-11-23T01:44:15.3967172Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3967282Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3967301Z 
2022-11-23T01:44:15.3967406Z OK (skipped=1)
2022-11-23T01:44:15.3967425Z 
2022-11-23T01:44:15.3967547Z Generating XML reports...
2022-11-23T01:44:15.3967993Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014100.xml
2022-11-23T01:44:15.3968360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3968535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3968908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3969102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3969121Z 
2022-11-23T01:44:15.3969212Z Running tests...
2022-11-23T01:44:15.3969471Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3969785Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3970049Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3970073Z 
2022-11-23T01:44:15.3970332Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3970443Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3970462Z 
2022-11-23T01:44:15.3970569Z OK (skipped=1)
2022-11-23T01:44:15.3970588Z 
2022-11-23T01:44:15.3970712Z Generating XML reports...
2022-11-23T01:44:15.3971155Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014102.xml
2022-11-23T01:44:15.3971846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3972032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3972415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3972610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3972630Z 
2022-11-23T01:44:15.3972742Z Running tests...
2022-11-23T01:44:15.3973004Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3973315Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3973569Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3973591Z 
2022-11-23T01:44:15.3973836Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3973948Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3973967Z 
2022-11-23T01:44:15.3974075Z OK (skipped=1)
2022-11-23T01:44:15.3974094Z 
2022-11-23T01:44:15.3974216Z Generating XML reports...
2022-11-23T01:44:15.3974657Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014105.xml
2022-11-23T01:44:15.3975025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3975272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3975668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3975861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3975880Z 
2022-11-23T01:44:15.3975974Z Running tests...
2022-11-23T01:44:15.3976239Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3976551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3976804Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3976824Z 
2022-11-23T01:44:15.3977086Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3977198Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3977217Z 
2022-11-23T01:44:15.3977326Z OK (skipped=1)
2022-11-23T01:44:15.3977348Z 
2022-11-23T01:44:15.3977471Z Generating XML reports...
2022-11-23T01:44:15.3977918Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014107.xml
2022-11-23T01:44:15.3978271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3978448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3978827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3979019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3979038Z 
2022-11-23T01:44:15.3979146Z Running tests...
2022-11-23T01:44:15.3979407Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3979717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3979973Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3980177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18410
2022-11-23T01:44:15.3980399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18411
2022-11-23T01:44:15.3980770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3981020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3981402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3981595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3981960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3982136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3982505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3982678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3982926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3983176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.3983576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3983971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.3984204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.3984477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.3984822Z STAGE:2022-11-23 01:41:14 18411:18411 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3985586Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:44:15.3985687Z   warnings.warn(
2022-11-23T01:44:15.3986022Z STAGE:2022-11-23 01:41:14 18410:18410 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3986780Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:44:15.3986898Z   warnings.warn(
2022-11-23T01:44:15.3987239Z STAGE:2022-11-23 01:41:14 18411:18411 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3987574Z STAGE:2022-11-23 01:41:14 18410:18410 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3987921Z STAGE:2022-11-23 01:41:14 18411:18411 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3988269Z STAGE:2022-11-23 01:41:14 18410:18410 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3988596Z STAGE:2022-11-23 01:41:15 18411:18411 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3988914Z STAGE:2022-11-23 01:41:15 18411:18411 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3989258Z STAGE:2022-11-23 01:41:15 18411:18411 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3989589Z STAGE:2022-11-23 01:41:15 18410:18410 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.3989922Z STAGE:2022-11-23 01:41:15 18410:18410 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.3990265Z STAGE:2022-11-23 01:41:15 18410:18410 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.3990368Z ok (5.728s)
2022-11-23T01:44:15.3990434Z 
2022-11-23T01:44:15.3990705Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3990820Z Ran 1 test in 5.728s
2022-11-23T01:44:15.3990840Z 
2022-11-23T01:44:15.3990931Z OK
2022-11-23T01:44:15.3990950Z 
2022-11-23T01:44:15.3991056Z Generating XML reports...
2022-11-23T01:44:15.3991507Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014109.xml
2022-11-23T01:44:15.3991880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3992059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3992441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3992634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3992653Z 
2022-11-23T01:44:15.3992766Z Running tests...
2022-11-23T01:44:15.3993023Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3993338Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3993584Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.3993604Z 
2022-11-23T01:44:15.3993859Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3993972Z Ran 1 test in 0.002s
2022-11-23T01:44:15.3993992Z 
2022-11-23T01:44:15.3994100Z OK (skipped=1)
2022-11-23T01:44:15.3994165Z 
2022-11-23T01:44:15.3994294Z Generating XML reports...
2022-11-23T01:44:15.3994747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014118.xml
2022-11-23T01:44:15.3995115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3995292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3995656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3995846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3995866Z 
2022-11-23T01:44:15.3995974Z Running tests...
2022-11-23T01:44:15.3996237Z ----------------------------------------------------------------------
2022-11-23T01:44:15.3996550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.3996824Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.3997045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18561
2022-11-23T01:44:15.3997262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18562
2022-11-23T01:44:15.3997634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3997793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3998170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3998362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3998722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.3998897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.3999267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.3999458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.3999704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.3999996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4000401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4000797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4001029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4001259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4001594Z STAGE:2022-11-23 01:41:25 18562:18562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4001915Z STAGE:2022-11-23 01:41:25 18561:18561 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4002249Z STAGE:2022-11-23 01:41:26 18561:18561 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4002588Z STAGE:2022-11-23 01:41:26 18562:18562 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4002918Z STAGE:2022-11-23 01:41:26 18561:18561 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4003263Z STAGE:2022-11-23 01:41:26 18562:18562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4003590Z STAGE:2022-11-23 01:41:26 18561:18561 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4003971Z STAGE:2022-11-23 01:41:26 18561:18561 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4004322Z STAGE:2022-11-23 01:41:26 18561:18561 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4004649Z STAGE:2022-11-23 01:41:26 18562:18562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4004983Z STAGE:2022-11-23 01:41:26 18562:18562 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4005324Z STAGE:2022-11-23 01:41:26 18562:18562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4005427Z ok (6.462s)
2022-11-23T01:44:15.4005447Z 
2022-11-23T01:44:15.4005692Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4005802Z Ran 1 test in 6.463s
2022-11-23T01:44:15.4005822Z 
2022-11-23T01:44:15.4005908Z OK
2022-11-23T01:44:15.4005928Z 
2022-11-23T01:44:15.4006050Z Generating XML reports...
2022-11-23T01:44:15.4006506Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014120.xml
2022-11-23T01:44:15.4006877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4007050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4007433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4007625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4007645Z 
2022-11-23T01:44:15.4007737Z Running tests...
2022-11-23T01:44:15.4007999Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4008314Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4008579Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4008796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18680
2022-11-23T01:44:15.4009016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18681
2022-11-23T01:44:15.4009387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4009618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4009973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4010149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4010527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4010721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4011097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4011285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4011751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4012003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4012414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4012794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4013024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4013252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4013426Z ok (5.659s)
2022-11-23T01:44:15.4013450Z 
2022-11-23T01:44:15.4013724Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4013838Z Ran 1 test in 5.660s
2022-11-23T01:44:15.4013857Z 
2022-11-23T01:44:15.4013949Z OK
2022-11-23T01:44:15.4013968Z 
2022-11-23T01:44:15.4014090Z Generating XML reports...
2022-11-23T01:44:15.4014530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014129.xml
2022-11-23T01:44:15.4014904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4015079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4015462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4015651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4015673Z 
2022-11-23T01:44:15.4015779Z Running tests...
2022-11-23T01:44:15.4016044Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4016356Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4016612Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.4016634Z 
2022-11-23T01:44:15.4016876Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4016989Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4017009Z 
2022-11-23T01:44:15.4017116Z OK (skipped=1)
2022-11-23T01:44:15.4017136Z 
2022-11-23T01:44:15.4017260Z Generating XML reports...
2022-11-23T01:44:15.4017706Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014137.xml
2022-11-23T01:44:15.4018077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4018254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4018634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4018826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4018923Z 
2022-11-23T01:44:15.4019019Z Running tests...
2022-11-23T01:44:15.4019282Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4019597Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4019850Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4020069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18828
2022-11-23T01:44:15.4020287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18829
2022-11-23T01:44:15.4020661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4020840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4021201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4021394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4021759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4021931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4022302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4022491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4022782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4023033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4023434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4023817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4024053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4024279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4024613Z STAGE:2022-11-23 01:41:45 18829:18829 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4024935Z STAGE:2022-11-23 01:41:45 18828:18828 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4025268Z STAGE:2022-11-23 01:41:45 18829:18829 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4025591Z STAGE:2022-11-23 01:41:45 18828:18828 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4026156Z STAGE:2022-11-23 01:41:45 18829:18829 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:41:45 18828:18828 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4026180Z 
2022-11-23T01:44:15.4026509Z STAGE:2022-11-23 01:41:45 18829:18829 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4026841Z STAGE:2022-11-23 01:41:45 18829:18829 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4027170Z STAGE:2022-11-23 01:41:45 18829:18829 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4027501Z STAGE:2022-11-23 01:41:45 18828:18828 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4027837Z STAGE:2022-11-23 01:41:45 18828:18828 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4028225Z STAGE:2022-11-23 01:41:45 18828:18828 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4028331Z ok (5.758s)
2022-11-23T01:44:15.4028350Z 
2022-11-23T01:44:15.4028673Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4028786Z Ran 1 test in 5.758s
2022-11-23T01:44:15.4028806Z 
2022-11-23T01:44:15.4028899Z OK
2022-11-23T01:44:15.4028918Z 
2022-11-23T01:44:15.4029025Z Generating XML reports...
2022-11-23T01:44:15.4029474Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014140.xml
2022-11-23T01:44:15.4029840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4030019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4030398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4030589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4030608Z 
2022-11-23T01:44:15.4030717Z Running tests...
2022-11-23T01:44:15.4030978Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4031296Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4031543Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4031765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18946
2022-11-23T01:44:15.4031985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18947
2022-11-23T01:44:15.4032401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4032582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4032962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4033154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4033516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4033673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4034044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4034226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4034470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4034718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4035114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4035510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4035744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4035972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4036288Z STAGE:2022-11-23 01:41:53 18947:18947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4036617Z STAGE:2022-11-23 01:41:53 18946:18946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4036950Z STAGE:2022-11-23 01:41:53 18947:18947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4037275Z STAGE:2022-11-23 01:41:53 18946:18946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4037622Z STAGE:2022-11-23 01:41:53 18947:18947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4037965Z STAGE:2022-11-23 01:41:53 18946:18946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4038347Z STAGE:2022-11-23 01:41:53 18947:18947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4038675Z STAGE:2022-11-23 01:41:53 18947:18947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4039018Z STAGE:2022-11-23 01:41:53 18947:18947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4039324Z STAGE:2022-11-23 01:41:53 18946:18946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4039657Z STAGE:2022-11-23 01:41:53 18946:18946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4040000Z STAGE:2022-11-23 01:41:53 18946:18946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4040102Z ok (5.752s)
2022-11-23T01:44:15.4040121Z 
2022-11-23T01:44:15.4040384Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4040497Z Ran 1 test in 5.752s
2022-11-23T01:44:15.4040517Z 
2022-11-23T01:44:15.4040609Z OK
2022-11-23T01:44:15.4040629Z 
2022-11-23T01:44:15.4040753Z Generating XML reports...
2022-11-23T01:44:15.4041184Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014148.xml
2022-11-23T01:44:15.4041556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4041730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4042161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4042360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4042379Z 
2022-11-23T01:44:15.4042489Z Running tests...
2022-11-23T01:44:15.4042754Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4043067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4043334Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.4043354Z 
2022-11-23T01:44:15.4043599Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4043711Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4043730Z 
2022-11-23T01:44:15.4043838Z OK (skipped=1)
2022-11-23T01:44:15.4043857Z 
2022-11-23T01:44:15.4043984Z Generating XML reports...
2022-11-23T01:44:15.4044436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014156.xml
2022-11-23T01:44:15.4044803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4044979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4045358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4045550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4045569Z 
2022-11-23T01:44:15.4045661Z Running tests...
2022-11-23T01:44:15.4045922Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4046236Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4046486Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.4046509Z 
2022-11-23T01:44:15.4046768Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4046881Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4046900Z 
2022-11-23T01:44:15.4047008Z OK (skipped=1)
2022-11-23T01:44:15.4047027Z 
2022-11-23T01:44:15.4047150Z Generating XML reports...
2022-11-23T01:44:15.4047598Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014159.xml
2022-11-23T01:44:15.4048008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4048184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4048565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4048756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4048775Z 
2022-11-23T01:44:15.4048887Z Running tests...
2022-11-23T01:44:15.4049154Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4049466Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4049726Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.4049749Z 
2022-11-23T01:44:15.4050008Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4050104Z Ran 1 test in 0.003s
2022-11-23T01:44:15.4050123Z 
2022-11-23T01:44:15.4050231Z OK (skipped=1)
2022-11-23T01:44:15.4050250Z 
2022-11-23T01:44:15.4050375Z Generating XML reports...
2022-11-23T01:44:15.4050819Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014201.xml
2022-11-23T01:44:15.4051186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4051409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4052040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4052235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4052255Z 
2022-11-23T01:44:15.4052351Z Running tests...
2022-11-23T01:44:15.4052612Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4052924Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4053187Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.4053207Z 
2022-11-23T01:44:15.4053463Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4053574Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4053593Z 
2022-11-23T01:44:15.4053704Z OK (skipped=1)
2022-11-23T01:44:15.4053723Z 
2022-11-23T01:44:15.4053846Z Generating XML reports...
2022-11-23T01:44:15.4054294Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014204.xml
2022-11-23T01:44:15.4054648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4054827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4055203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4055394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4055414Z 
2022-11-23T01:44:15.4055522Z Running tests...
2022-11-23T01:44:15.4055783Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4056100Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4056347Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4056551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19196
2022-11-23T01:44:15.4056770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19197
2022-11-23T01:44:15.4057142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4057410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4057793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4057985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4058349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4058526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4058897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4059071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4059319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4059566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4059967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4060361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4060591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4060882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4061235Z STAGE:2022-11-23 01:42:11 19196:19196 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4061558Z STAGE:2022-11-23 01:42:11 19197:19197 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4061876Z STAGE:2022-11-23 01:42:12 19196:19196 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4062229Z STAGE:2022-11-23 01:42:12 19196:19196 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4062560Z STAGE:2022-11-23 01:42:12 19197:19197 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4062905Z STAGE:2022-11-23 01:42:12 19197:19197 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4063230Z STAGE:2022-11-23 01:42:12 19197:19197 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4063555Z STAGE:2022-11-23 01:42:12 19196:19196 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4063889Z STAGE:2022-11-23 01:42:12 19197:19197 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4064233Z STAGE:2022-11-23 01:42:12 19197:19197 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4064570Z STAGE:2022-11-23 01:42:12 19196:19196 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4064899Z STAGE:2022-11-23 01:42:12 19196:19196 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4065002Z ok (6.546s)
2022-11-23T01:44:15.4065022Z 
2022-11-23T01:44:15.4065284Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4065397Z Ran 1 test in 6.546s
2022-11-23T01:44:15.4065416Z 
2022-11-23T01:44:15.4065506Z OK
2022-11-23T01:44:15.4065525Z 
2022-11-23T01:44:15.4065647Z Generating XML reports...
2022-11-23T01:44:15.4066098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014206.xml
2022-11-23T01:44:15.4066468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4066628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4067071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4067265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4067285Z 
2022-11-23T01:44:15.4067395Z Running tests...
2022-11-23T01:44:15.4067658Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4067972Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4068241Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4068461Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19319
2022-11-23T01:44:15.4068678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19320
2022-11-23T01:44:15.4069033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4069216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4069592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4069784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4070145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4070319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4070735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4070932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4071161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4071408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4071817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4072213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4072443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4072670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4073009Z STAGE:2022-11-23 01:42:20 19319:19319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4073333Z STAGE:2022-11-23 01:42:20 19320:19320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4073669Z STAGE:2022-11-23 01:42:21 19319:19319 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4074003Z STAGE:2022-11-23 01:42:21 19319:19319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4074336Z STAGE:2022-11-23 01:42:21 19320:19320 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4074680Z STAGE:2022-11-23 01:42:21 19320:19320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4075005Z STAGE:2022-11-23 01:42:21 19319:19319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4075329Z STAGE:2022-11-23 01:42:21 19320:19320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4075660Z STAGE:2022-11-23 01:42:21 19320:19320 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4076003Z STAGE:2022-11-23 01:42:21 19320:19320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4076336Z STAGE:2022-11-23 01:42:21 19319:19319 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4076744Z STAGE:2022-11-23 01:42:21 19319:19319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4076831Z ok (6.557s)
2022-11-23T01:44:15.4076850Z 
2022-11-23T01:44:15.4077115Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4077227Z Ran 1 test in 6.557s
2022-11-23T01:44:15.4077246Z 
2022-11-23T01:44:15.4077340Z OK
2022-11-23T01:44:15.4077359Z 
2022-11-23T01:44:15.4077483Z Generating XML reports...
2022-11-23T01:44:15.4077938Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014215.xml
2022-11-23T01:44:15.4078308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4078484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4078848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4079043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4079063Z 
2022-11-23T01:44:15.4079170Z Running tests...
2022-11-23T01:44:15.4079430Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4079742Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4080010Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.4080030Z 
2022-11-23T01:44:15.4080334Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4080453Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4080472Z 
2022-11-23T01:44:15.4080580Z OK (skipped=1)
2022-11-23T01:44:15.4080599Z 
2022-11-23T01:44:15.4080704Z Generating XML reports...
2022-11-23T01:44:15.4081152Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014224.xml
2022-11-23T01:44:15.4081528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4081705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4082081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4082271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4082291Z 
2022-11-23T01:44:15.4082398Z Running tests...
2022-11-23T01:44:15.4082659Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4082971Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4083217Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T01:44:15.4083236Z 
2022-11-23T01:44:15.4083500Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4083612Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4083632Z 
2022-11-23T01:44:15.4083740Z OK (skipped=1)
2022-11-23T01:44:15.4083758Z 
2022-11-23T01:44:15.4083881Z Generating XML reports...
2022-11-23T01:44:15.4084325Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014227.xml
2022-11-23T01:44:15.4084693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4084870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4085250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4085427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4085447Z 
2022-11-23T01:44:15.4085555Z Running tests...
2022-11-23T01:44:15.4085892Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4086205Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4086587Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T01:44:15.4086606Z 
2022-11-23T01:44:15.4086864Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4086977Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4086996Z 
2022-11-23T01:44:15.4087102Z OK (skipped=1)
2022-11-23T01:44:15.4087125Z 
2022-11-23T01:44:15.4087230Z Generating XML reports...
2022-11-23T01:44:15.4087673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014229.xml
2022-11-23T01:44:15.4088041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4088220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4088596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4088787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4088806Z 
2022-11-23T01:44:15.4088914Z Running tests...
2022-11-23T01:44:15.4089174Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4089485Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4089780Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:44:15.4089820Z 
2022-11-23T01:44:15.4090066Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4090178Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4090197Z 
2022-11-23T01:44:15.4090304Z OK (skipped=1)
2022-11-23T01:44:15.4090327Z 
2022-11-23T01:44:15.4090450Z Generating XML reports...
2022-11-23T01:44:15.4090900Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014231.xml
2022-11-23T01:44:15.4091268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4091657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4092049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4092230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4092250Z 
2022-11-23T01:44:15.4092357Z Running tests...
2022-11-23T01:44:15.4092619Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4092932Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4093216Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T01:44:15.4093236Z 
2022-11-23T01:44:15.4093494Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4093606Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4093625Z 
2022-11-23T01:44:15.4093732Z OK (skipped=1)
2022-11-23T01:44:15.4093750Z 
2022-11-23T01:44:15.4093872Z Generating XML reports...
2022-11-23T01:44:15.4094305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014234.xml
2022-11-23T01:44:15.4094672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4094846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4095223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4095502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4095521Z 
2022-11-23T01:44:15.4095628Z Running tests...
2022-11-23T01:44:15.4095895Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4096208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4096503Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T01:44:15.4096541Z 
2022-11-23T01:44:15.4096785Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4096896Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4096915Z 
2022-11-23T01:44:15.4097022Z OK (skipped=1)
2022-11-23T01:44:15.4097041Z 
2022-11-23T01:44:15.4097165Z Generating XML reports...
2022-11-23T01:44:15.4097607Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014236.xml
2022-11-23T01:44:15.4097975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4098149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4098523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4098696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4098731Z 
2022-11-23T01:44:15.4098822Z Running tests...
2022-11-23T01:44:15.4099148Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4099470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4099777Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T01:44:15.4099800Z 
2022-11-23T01:44:15.4100057Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4100167Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4100187Z 
2022-11-23T01:44:15.4100292Z OK (skipped=1)
2022-11-23T01:44:15.4100311Z 
2022-11-23T01:44:15.4100435Z Generating XML reports...
2022-11-23T01:44:15.4100867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014239.xml
2022-11-23T01:44:15.4101241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4101417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4101795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4101989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4102008Z 
2022-11-23T01:44:15.4102119Z Running tests...
2022-11-23T01:44:15.4102381Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4102693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4102982Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:44:15.4103003Z 
2022-11-23T01:44:15.4103244Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4103353Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4103372Z 
2022-11-23T01:44:15.4103483Z OK (skipped=1)
2022-11-23T01:44:15.4103502Z 
2022-11-23T01:44:15.4103623Z Generating XML reports...
2022-11-23T01:44:15.4104067Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014241.xml
2022-11-23T01:44:15.4104432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4104664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4105044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4105233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4105252Z 
2022-11-23T01:44:15.4105343Z Running tests...
2022-11-23T01:44:15.4105604Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4105917Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4106168Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4106389Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19706
2022-11-23T01:44:15.4106609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19707
2022-11-23T01:44:15.4106975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4107151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4107514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4107703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4108064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4108287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4108667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4108860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4109106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4109349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4109749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4110128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4110360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4110593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4110696Z ok (5.253s)
2022-11-23T01:44:15.4110716Z 
2022-11-23T01:44:15.4110981Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4111093Z Ran 1 test in 5.253s
2022-11-23T01:44:15.4111112Z 
2022-11-23T01:44:15.4111207Z OK
2022-11-23T01:44:15.4111227Z 
2022-11-23T01:44:15.4111350Z Generating XML reports...
2022-11-23T01:44:15.4111785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014243.xml
2022-11-23T01:44:15.4112159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4112334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4112712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4112901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4112920Z 
2022-11-23T01:44:15.4113031Z Running tests...
2022-11-23T01:44:15.4113296Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4113609Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4113948Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4114152Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19824
2022-11-23T01:44:15.4114376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19825
2022-11-23T01:44:15.4114749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4114925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4115305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4115495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4115856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4116032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4116389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4116578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4116822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4117068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4117550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4117957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4118189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4118421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4118756Z STAGE:2022-11-23 01:42:56 19824:19824 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4119063Z STAGE:2022-11-23 01:42:56 19825:19825 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4119400Z STAGE:2022-11-23 01:42:56 19825:19825 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4119726Z STAGE:2022-11-23 01:42:56 19824:19824 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4120075Z STAGE:2022-11-23 01:42:56 19825:19825 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4120418Z STAGE:2022-11-23 01:42:56 19824:19824 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4120520Z ok (5.742s)
2022-11-23T01:44:15.4120539Z 
2022-11-23T01:44:15.4120802Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4120920Z Ran 1 test in 5.742s
2022-11-23T01:44:15.4120939Z 
2022-11-23T01:44:15.4121032Z OK
2022-11-23T01:44:15.4121051Z 
2022-11-23T01:44:15.4121158Z Generating XML reports...
2022-11-23T01:44:15.4121608Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014251.xml
2022-11-23T01:44:15.4121977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4122156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4122535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4122726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4122746Z 
2022-11-23T01:44:15.4122854Z Running tests...
2022-11-23T01:44:15.4123113Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4123470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4123747Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4123966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19946
2022-11-23T01:44:15.4124185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19947
2022-11-23T01:44:15.4124558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4124732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4125108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4125299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4125664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4125821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4126191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4126380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4126628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4126922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4127333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4127729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4127962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4128238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4128562Z STAGE:2022-11-23 01:43:04 19946:19946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4128885Z STAGE:2022-11-23 01:43:04 19947:19947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:44:15.4129223Z STAGE:2022-11-23 01:43:05 19947:19947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4129560Z STAGE:2022-11-23 01:43:05 19946:19946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:44:15.4129906Z STAGE:2022-11-23 01:43:05 19947:19947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4130251Z STAGE:2022-11-23 01:43:05 19946:19946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:44:15.4130355Z ok (5.761s)
2022-11-23T01:44:15.4130375Z 
2022-11-23T01:44:15.4130638Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4130733Z Ran 1 test in 5.761s
2022-11-23T01:44:15.4130768Z 
2022-11-23T01:44:15.4130843Z OK
2022-11-23T01:44:15.4130862Z 
2022-11-23T01:44:15.4130985Z Generating XML reports...
2022-11-23T01:44:15.4131649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014300.xml
2022-11-23T01:44:15.4132045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4132220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4132599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4132790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4132887Z 
2022-11-23T01:44:15.4133000Z Running tests...
2022-11-23T01:44:15.4133254Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4133568Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4133856Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:44:15.4133876Z 
2022-11-23T01:44:15.4134135Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4134249Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4134269Z 
2022-11-23T01:44:15.4134378Z OK (skipped=1)
2022-11-23T01:44:15.4134397Z 
2022-11-23T01:44:15.4134521Z Generating XML reports...
2022-11-23T01:44:15.4134969Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014308.xml
2022-11-23T01:44:15.4135335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4135497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4135874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4136062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4136081Z 
2022-11-23T01:44:15.4136190Z Running tests...
2022-11-23T01:44:15.4136451Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4136830Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4137114Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:44:15.4137134Z 
2022-11-23T01:44:15.4137393Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4137492Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4137527Z 
2022-11-23T01:44:15.4137618Z OK (skipped=1)
2022-11-23T01:44:15.4137636Z 
2022-11-23T01:44:15.4137760Z Generating XML reports...
2022-11-23T01:44:15.4138208Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014310.xml
2022-11-23T01:44:15.4138577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4138752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4139133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4139325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4139345Z 
2022-11-23T01:44:15.4139453Z Running tests...
2022-11-23T01:44:15.4139698Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4140013Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4140312Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:44:15.4140332Z 
2022-11-23T01:44:15.4140594Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4140707Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4140726Z 
2022-11-23T01:44:15.4140833Z OK (skipped=1)
2022-11-23T01:44:15.4140852Z 
2022-11-23T01:44:15.4140978Z Generating XML reports...
2022-11-23T01:44:15.4141428Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014313.xml
2022-11-23T01:44:15.4141794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4141954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4142395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4142587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4142606Z 
2022-11-23T01:44:15.4142712Z Running tests...
2022-11-23T01:44:15.4142972Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4143283Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4143581Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T01:44:15.4143601Z 
2022-11-23T01:44:15.4143862Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4143971Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4143990Z 
2022-11-23T01:44:15.4144080Z OK (skipped=1)
2022-11-23T01:44:15.4144099Z 
2022-11-23T01:44:15.4144224Z Generating XML reports...
2022-11-23T01:44:15.4144670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014315.xml
2022-11-23T01:44:15.4145038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4145212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4145586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4145821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4145843Z 
2022-11-23T01:44:15.4145955Z Running tests...
2022-11-23T01:44:15.4146203Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4146516Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4146798Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-11-23T01:44:15.4146822Z 
2022-11-23T01:44:15.4147084Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4147196Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4147215Z 
2022-11-23T01:44:15.4147349Z OK (skipped=1)
2022-11-23T01:44:15.4147376Z 
2022-11-23T01:44:15.4147601Z Generating XML reports...
2022-11-23T01:44:15.4148124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014317.xml
2022-11-23T01:44:15.4148499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4148658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4149036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4149226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4149250Z 
2022-11-23T01:44:15.4149358Z Running tests...
2022-11-23T01:44:15.4149620Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4150032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4150334Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-11-23T01:44:15.4150355Z 
2022-11-23T01:44:15.4150622Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4150734Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4150753Z 
2022-11-23T01:44:15.4150842Z OK (skipped=1)
2022-11-23T01:44:15.4150878Z 
2022-11-23T01:44:15.4150983Z Generating XML reports...
2022-11-23T01:44:15.4151431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014320.xml
2022-11-23T01:44:15.4151888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4152066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4152443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4152635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4152655Z 
2022-11-23T01:44:15.4152762Z Running tests...
2022-11-23T01:44:15.4153026Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4153323Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4153589Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4153811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20266
2022-11-23T01:44:15.4154035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20267
2022-11-23T01:44:15.4154405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4154581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4154959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4155148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4155543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4155724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4156102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4156290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4156541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4156786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4157184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4157581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4157815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4158028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4158128Z ok (6.384s)
2022-11-23T01:44:15.4158147Z 
2022-11-23T01:44:15.4158414Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4158527Z Ran 1 test in 6.384s
2022-11-23T01:44:15.4158546Z 
2022-11-23T01:44:15.4158639Z OK
2022-11-23T01:44:15.4158658Z 
2022-11-23T01:44:15.4158783Z Generating XML reports...
2022-11-23T01:44:15.4159232Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014322.xml
2022-11-23T01:44:15.4159603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4159778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4160142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4160333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4160353Z 
2022-11-23T01:44:15.4160460Z Running tests...
2022-11-23T01:44:15.4160723Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4161092Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4161366Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T01:44:15.4161386Z 
2022-11-23T01:44:15.4161644Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4161757Z Ran 1 test in 0.002s
2022-11-23T01:44:15.4161776Z 
2022-11-23T01:44:15.4161866Z OK (skipped=1)
2022-11-23T01:44:15.4161902Z 
2022-11-23T01:44:15.4162007Z Generating XML reports...
2022-11-23T01:44:15.4162458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014331.xml
2022-11-23T01:44:15.4162829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4163006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4163390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4163582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4163601Z 
2022-11-23T01:44:15.4163709Z Running tests...
2022-11-23T01:44:15.4163968Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4164263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4164560Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4164789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20418
2022-11-23T01:44:15.4165009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20419
2022-11-23T01:44:15.4165384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4165562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4165938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4166129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4166475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4166646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4167027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4167220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4167467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4167711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4168115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4168507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4168738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4168950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4169212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjk4hkkud
2022-11-23T01:44:15.4169482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjk4hkkud/_remote_module_non_scriptable.py
2022-11-23T01:44:15.4169737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgeabz2av
2022-11-23T01:44:15.4170005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgeabz2av/_remote_module_non_scriptable.py
2022-11-23T01:44:15.4170166Z ok (5.633s)
2022-11-23T01:44:15.4170185Z 
2022-11-23T01:44:15.4170456Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4170569Z Ran 1 test in 5.633s
2022-11-23T01:44:15.4170588Z 
2022-11-23T01:44:15.4170681Z OK
2022-11-23T01:44:15.4170701Z 
2022-11-23T01:44:15.4170807Z Generating XML reports...
2022-11-23T01:44:15.4171255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014333.xml
2022-11-23T01:44:15.4171813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4171997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4172376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4172570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4172589Z 
2022-11-23T01:44:15.4172697Z Running tests...
2022-11-23T01:44:15.4172957Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4173254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4173544Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4173763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20533
2022-11-23T01:44:15.4174056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20534
2022-11-23T01:44:15.4174442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4174618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4174995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4175192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4175557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4175713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4176091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4176282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4176529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4176772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4177171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4177570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4177802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4178014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4178271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hn5w6v8
2022-11-23T01:44:15.4178542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hn5w6v8/_remote_module_non_scriptable.py
2022-11-23T01:44:15.4178798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthk3gd_s
2022-11-23T01:44:15.4179069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthk3gd_s/_remote_module_non_scriptable.py
2022-11-23T01:44:15.4179846Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:44:15.4180701Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:44:15.4180810Z ok (6.157s)
2022-11-23T01:44:15.4180829Z 
2022-11-23T01:44:15.4181100Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4181214Z Ran 1 test in 6.157s
2022-11-23T01:44:15.4181233Z 
2022-11-23T01:44:15.4181327Z OK
2022-11-23T01:44:15.4181346Z 
2022-11-23T01:44:15.4181470Z Generating XML reports...
2022-11-23T01:44:15.4181923Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014342.xml
2022-11-23T01:44:15.4182320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4182509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4182895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4183087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4183110Z 
2022-11-23T01:44:15.4183217Z Running tests...
2022-11-23T01:44:15.4183481Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4183796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4184076Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4184280Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20652
2022-11-23T01:44:15.4184503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20653
2022-11-23T01:44:15.4184868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4185044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4185421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4185617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4185982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4186153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4186526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4186702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4186952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4187197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4187596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4188054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4188285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4188515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4188755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.4189000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.4189381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.4189771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.4190016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:44:15.4190254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:44:15.4190641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.4191033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.4191339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpav4uajw_
2022-11-23T01:44:15.4191601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt4e_o5m
2022-11-23T01:44:15.4191869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpav4uajw_/_remote_module_non_scriptable.py
2022-11-23T01:44:15.4192113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt4e_o5m/_remote_module_non_scriptable.py
2022-11-23T01:44:15.4192220Z ok (10.626s)
2022-11-23T01:44:15.4192239Z 
2022-11-23T01:44:15.4192508Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4192620Z Ran 1 test in 10.626s
2022-11-23T01:44:15.4192639Z 
2022-11-23T01:44:15.4192733Z OK
2022-11-23T01:44:15.4192753Z 
2022-11-23T01:44:15.4192877Z Generating XML reports...
2022-11-23T01:44:15.4193328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014350.xml
2022-11-23T01:44:15.4193701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4193861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4194243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4194432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4194455Z 
2022-11-23T01:44:15.4194564Z Running tests...
2022-11-23T01:44:15.4194825Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4195140Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T01:44:15.4195430Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:44:15.4195649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20774
2022-11-23T01:44:15.4195870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20775
2022-11-23T01:44:15.4196225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4196401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4196783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4197031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4197396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:44:15.4197569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:44:15.4197940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:44:15.4198136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:44:15.4198367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:44:15.4198613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:44:15.4199012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4199410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:44:15.4199640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:44:15.4199873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:44:15.4200109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:44:15.4200395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:44:15.4200804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.4201179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:44:15.4201421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:44:15.4201665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:44:15.4202056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.4202447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:44:15.4202706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwgxczeq
2022-11-23T01:44:15.4202979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwgxczeq/_remote_module_non_scriptable.py
2022-11-23T01:44:15.4203230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpac3ijlmn
2022-11-23T01:44:15.4203497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpac3ijlmn/_remote_module_non_scriptable.py
2022-11-23T01:44:15.4203586Z ok (10.540s)
2022-11-23T01:44:15.4203620Z 
2022-11-23T01:44:15.4203868Z ----------------------------------------------------------------------
2022-11-23T01:44:15.4203982Z Ran 1 test in 10.540s
2022-11-23T01:44:15.4204001Z 
2022-11-23T01:44:15.4204096Z OK
2022-11-23T01:44:15.4204115Z 
2022-11-23T01:44:15.4204237Z Generating XML reports...
2022-11-23T01:44:15.4204687Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014404.xml
2022-11-23T01:44:15.4204707Z 
2022-11-23T01:44:15.4205156Z ##[endgroup]
2022-11-23T01:44:15.4205631Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_tbddl287)
2022-11-23T01:44:15.4205651Z 
2022-11-23T01:44:15.4205848Z Running distributed tests for the nccl backend with file init_method in shard 2 of 3
2022-11-23T01:44:15.4206357Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:44:15.222000]
2022-11-23T02:08:16.1243935Z 
2022-11-23T02:08:16.1246693Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T02:08:16.1247657Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_07am423u)
2022-11-23T02:08:16.1248154Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:08:16.1292657Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T02:08:16.1354290Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1355216Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1355970Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1359391Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1360193Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1361052Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1362067Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1362915Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1363848Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1364797Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1365723Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1366699Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1367625Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1368583Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1369396Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1370311Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1371164Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1372370Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1373121Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1374054Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1374972Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1375789Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1376460Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1377144Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1377943Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1378642Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1379361Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1380176Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1380856Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1381547Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1382309Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1383015Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1383662Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1384397Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1385176Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1385918Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1386600Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1387363Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1388121Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1388870Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1389659Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1390423Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1391159Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1391962Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1392698Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1393606Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1394369Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1395137Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1395902Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1396633Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1397412Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1398139Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1398825Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1399580Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1400318Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1401030Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1401733Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1402432Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1403112Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1403831Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1404558Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1405390Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1406049Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1406647Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1407307Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1408016Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1408753Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1409615Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1410120Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1410491Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1410889Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1411286Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1412260Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1412666Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1413060Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1413440Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1413804Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1414198Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1414601Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1414989Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1417450Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1417908Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1418315Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1418740Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1419180Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1419628Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1420069Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1420664Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1421116Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1421562Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1421983Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1422421Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1422865Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1423300Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1423763Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1424228Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1424685Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1425126Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1425553Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1425943Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1426307Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1426674Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1427041Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1427477Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1427890Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1428281Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1428665Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1429052Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1429465Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1429864Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1430244Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1430650Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1431075Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1431473Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1431892Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1432308Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1432723Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1433142Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1433574Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1433989Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1434358Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1434732Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1435122Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1435515Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1435889Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1436289Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1436746Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1437245Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1437701Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1438179Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1438584Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1439021Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1439472Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1439941Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1440373Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1440797Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1441242Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1441636Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1442010Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1442408Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1442826Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1443224Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1443663Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1444104Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1444507Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1444970Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1445450Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1445953Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1446504Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1447121Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1447727Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1448334Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1448928Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1449531Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1450131Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1450737Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1451269Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1452301Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1452757Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1453127Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1453525Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1453931Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1454330Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1454840Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1455285Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1455748Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1456200Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1456617Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1457010Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1457431Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1457853Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1458288Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1458705Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1459106Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1459532Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1459964Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1460372Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1460780Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1461191Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1461676Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1462078Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1462500Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1462961Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1463383Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1463787Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1464171Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1464576Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1464982Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1465400Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1465782Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1466134Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1466511Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1466892Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1467254Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1467628Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1468025Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1468395Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1468762Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1469121Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1469501Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1469884Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1470278Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1470652Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1470984Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1471363Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1471768Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1472166Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1472682Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1473134Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1473550Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1473955Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1474393Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1474824Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1475233Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1475650Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1476067Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1476476Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1476872Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1477270Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1477666Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1478123Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1478615Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1479132Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1479597Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1480042Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1480498Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1480929Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1481338Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1481766Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1482204Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1482640Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1483120Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1483638Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1484124Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1484526Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1484922Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1485331Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1485722Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1486110Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1486492Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1486882Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1487253Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1487624Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1487991Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1488351Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1488733Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1489130Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1489521Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1489960Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1490333Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1490725Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1491092Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1491856Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1492343Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1492713Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1493090Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1493475Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1493853Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1494233Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1494617Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1494976Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1495356Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1495778Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1496223Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1496638Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1497034Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1497521Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1497945Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1498361Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1498761Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1499181Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1499602Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1500016Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1500417Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1500807Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1501210Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1501598Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1502002Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1502453Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1502897Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1503685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1504126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1504704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1505173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1505406Z 
2022-11-23T02:08:16.1505516Z Running tests...
2022-11-23T02:08:16.1505907Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1506442Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1507029Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1507579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20929
2022-11-23T02:08:16.1508121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20930
2022-11-23T02:08:16.1508738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1509191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1509748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1510214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1510794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1511222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1511792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1512256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1512712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1513194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1513851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1514597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1515135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1515592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1516113Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:08:16.1516952Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:08:16.1517627Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:08:16.1518429Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:08:16.1519093Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:08:16.1519908Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:08:16.1520567Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:08:16.1521377Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:08:16.1522029Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:08:16.1522841Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:08:16.1523508Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:08:16.1524305Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:08:16.1524959Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:08:16.1525859Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:08:16.1526515Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:08:16.1527311Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:08:16.1527789Z ok (5.751s)
2022-11-23T02:08:16.1527939Z 
2022-11-23T02:08:16.1528210Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1528540Z Ran 1 test in 5.751s
2022-11-23T02:08:16.1528685Z 
2022-11-23T02:08:16.1528778Z OK
2022-11-23T02:08:16.1528913Z 
2022-11-23T02:08:16.1529037Z Generating XML reports...
2022-11-23T02:08:16.1529649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014419.xml
2022-11-23T02:08:16.1530358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1530808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1531388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1532430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1532648Z 
2022-11-23T02:08:16.1532758Z Running tests...
2022-11-23T02:08:16.1533262Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1533812Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1534328Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-11-23T02:08:16.1534646Z 
2022-11-23T02:08:16.1534913Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1535242Z Ran 1 test in 0.004s
2022-11-23T02:08:16.1535406Z 
2022-11-23T02:08:16.1535514Z OK (skipped=1)
2022-11-23T02:08:16.1535652Z 
2022-11-23T02:08:16.1535777Z Generating XML reports...
2022-11-23T02:08:16.1536386Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014427.xml
2022-11-23T02:08:16.1537107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1537549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1538126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1538595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1538826Z 
2022-11-23T02:08:16.1538935Z Running tests...
2022-11-23T02:08:16.1539326Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1539859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1540369Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1540842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21077
2022-11-23T02:08:16.1541289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21078
2022-11-23T02:08:16.1541897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1542348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1542904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1543371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1544036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1544478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1545032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1545559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1546017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1546501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1547162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1547846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1548372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1548826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1549164Z ok (4.201s)
2022-11-23T02:08:16.1549313Z 
2022-11-23T02:08:16.1549582Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1549893Z Ran 1 test in 4.201s
2022-11-23T02:08:16.1550055Z 
2022-11-23T02:08:16.1550149Z OK
2022-11-23T02:08:16.1550285Z 
2022-11-23T02:08:16.1550463Z Generating XML reports...
2022-11-23T02:08:16.1551085Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014430.xml
2022-11-23T02:08:16.1551789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1552241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1552822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1553294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1553507Z 
2022-11-23T02:08:16.1553615Z Running tests...
2022-11-23T02:08:16.1554016Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1554547Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1555064Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1556121Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2022-11-23T02:08:16.1556652Z 
2022-11-23T02:08:16.1556916Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1557242Z Ran 1 test in 1.634s
2022-11-23T02:08:16.1557402Z 
2022-11-23T02:08:16.1557511Z OK (skipped=1)
2022-11-23T02:08:16.1557649Z 
2022-11-23T02:08:16.1557774Z Generating XML reports...
2022-11-23T02:08:16.1558379Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014436.xml
2022-11-23T02:08:16.1559097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1559532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1560105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1560574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1560893Z 
2022-11-23T02:08:16.1561002Z Running tests...
2022-11-23T02:08:16.1561392Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1561921Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1562468Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T02:08:16.1562794Z 
2022-11-23T02:08:16.1563039Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1563367Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1563532Z 
2022-11-23T02:08:16.1563641Z OK (skipped=1)
2022-11-23T02:08:16.1563795Z 
2022-11-23T02:08:16.1563919Z Generating XML reports...
2022-11-23T02:08:16.1564510Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014441.xml
2022-11-23T02:08:16.1565224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1565680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1566258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1566711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1566944Z 
2022-11-23T02:08:16.1567052Z Running tests...
2022-11-23T02:08:16.1567455Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1568020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1568600Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T02:08:16.1568943Z 
2022-11-23T02:08:16.1569204Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1569534Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1569676Z 
2022-11-23T02:08:16.1569783Z OK (skipped=1)
2022-11-23T02:08:16.1569935Z 
2022-11-23T02:08:16.1570058Z Generating XML reports...
2022-11-23T02:08:16.1570662Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014443.xml
2022-11-23T02:08:16.1571359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1572317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1572909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1573375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1573587Z 
2022-11-23T02:08:16.1573696Z Running tests...
2022-11-23T02:08:16.1574100Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1574639Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1575169Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1575699Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21282
2022-11-23T02:08:16.1576146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21283
2022-11-23T02:08:16.1576751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1577187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1577762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1578230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1578808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1579340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1579914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1580378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1580816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1581320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1581976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1582668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1583177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1583648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1584149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12y87tns
2022-11-23T02:08:16.1584692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12y87tns/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1585202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbuhxql69
2022-11-23T02:08:16.1585806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbuhxql69/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1586731Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1587287Z   warnings.warn(
2022-11-23T02:08:16.1588031Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1588577Z   warnings.warn(
2022-11-23T02:08:16.1588949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1589420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1589902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1590382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1590856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1591314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1591792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1592261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1592714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1593187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1593652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1594122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1594453Z ok (6.538s)
2022-11-23T02:08:16.1594599Z 
2022-11-23T02:08:16.1594874Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1595203Z Ran 1 test in 6.538s
2022-11-23T02:08:16.1595365Z 
2022-11-23T02:08:16.1595440Z OK
2022-11-23T02:08:16.1595573Z 
2022-11-23T02:08:16.1595761Z Generating XML reports...
2022-11-23T02:08:16.1596374Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014445.xml
2022-11-23T02:08:16.1597089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1597523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1598097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1598568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1598796Z 
2022-11-23T02:08:16.1598887Z Running tests...
2022-11-23T02:08:16.1599292Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1599821Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1600385Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1600913Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21401
2022-11-23T02:08:16.1601363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21402
2022-11-23T02:08:16.1601970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1602401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1603024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1603564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1604144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1604572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1605149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1605615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1606070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1606550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1607211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1607897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1608404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1608878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1609381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp168gcbl7
2022-11-23T02:08:16.1609923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp168gcbl7/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1610437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplv7g4z89
2022-11-23T02:08:16.1610968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplv7g4z89/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1612403Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1612960Z   warnings.warn(
2022-11-23T02:08:16.1613710Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1614352Z   warnings.warn(
2022-11-23T02:08:16.1614731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1615221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1615683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1616166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1616515Z ok (5.642s)
2022-11-23T02:08:16.1616663Z 
2022-11-23T02:08:16.1616922Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1617250Z Ran 1 test in 5.643s
2022-11-23T02:08:16.1617411Z 
2022-11-23T02:08:16.1617506Z OK
2022-11-23T02:08:16.1617640Z 
2022-11-23T02:08:16.1617764Z Generating XML reports...
2022-11-23T02:08:16.1618362Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014454.xml
2022-11-23T02:08:16.1619078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1619531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1620091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1620631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1620872Z 
2022-11-23T02:08:16.1620982Z Running tests...
2022-11-23T02:08:16.1621389Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1621899Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1622472Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1623022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21520
2022-11-23T02:08:16.1623454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21521
2022-11-23T02:08:16.1624062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1624512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1625088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1625542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1626120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1626564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1627123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1627591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1628043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1628539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1629182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1629877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1630397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1630868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1631426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6h59ujh
2022-11-23T02:08:16.1631968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6h59ujh/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1632585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcycdsjf
2022-11-23T02:08:16.1633279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcycdsjf/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1634438Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1635000Z   warnings.warn(
2022-11-23T02:08:16.1635761Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1636304Z   warnings.warn(
2022-11-23T02:08:16.1636662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1637149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1637628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1638089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1638627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1639109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1639581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1640040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1640390Z ok (5.835s)
2022-11-23T02:08:16.1640536Z 
2022-11-23T02:08:16.1640811Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1641120Z Ran 1 test in 5.835s
2022-11-23T02:08:16.1641281Z 
2022-11-23T02:08:16.1641375Z OK
2022-11-23T02:08:16.1641508Z 
2022-11-23T02:08:16.1641632Z Generating XML reports...
2022-11-23T02:08:16.1642223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014503.xml
2022-11-23T02:08:16.1642941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1643392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1643967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1644422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1644659Z 
2022-11-23T02:08:16.1644768Z Running tests...
2022-11-23T02:08:16.1645169Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1645701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1646274Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1646836Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21639
2022-11-23T02:08:16.1647285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21640
2022-11-23T02:08:16.1647878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1648327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1648899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1649449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1650014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1650465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1651033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1651937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1652553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1653053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1653718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1654393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1654911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1655384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1655884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzakm85dc
2022-11-23T02:08:16.1656501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzakm85dc/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1657046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fq8srd7
2022-11-23T02:08:16.1657580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fq8srd7/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1658490Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1659027Z   warnings.warn(
2022-11-23T02:08:16.1659776Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1660316Z   warnings.warn(
2022-11-23T02:08:16.1660679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1661170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1661520Z ok (5.755s)
2022-11-23T02:08:16.1661668Z 
2022-11-23T02:08:16.1661938Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1662248Z Ran 1 test in 5.755s
2022-11-23T02:08:16.1662412Z 
2022-11-23T02:08:16.1662505Z OK
2022-11-23T02:08:16.1662639Z 
2022-11-23T02:08:16.1662764Z Generating XML reports...
2022-11-23T02:08:16.1663356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014511.xml
2022-11-23T02:08:16.1664075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1664527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1665104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1665561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1665790Z 
2022-11-23T02:08:16.1665904Z Running tests...
2022-11-23T02:08:16.1666303Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1666813Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1667482Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1668043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21758
2022-11-23T02:08:16.1668492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21759
2022-11-23T02:08:16.1669085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1669537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1670111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1670583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1671144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1671592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1672162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1672612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1673070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1673613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1674281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1674950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1675470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1675944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1676442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmak_0x0g
2022-11-23T02:08:16.1676963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmak_0x0g/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1677493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppedyr8f9
2022-11-23T02:08:16.1678030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppedyr8f9/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1678920Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1679467Z   warnings.warn(
2022-11-23T02:08:16.1680221Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1680762Z   warnings.warn(
2022-11-23T02:08:16.1681122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1681609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1682096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1682577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1682913Z ok (6.206s)
2022-11-23T02:08:16.1683060Z 
2022-11-23T02:08:16.1683330Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1683657Z Ran 1 test in 6.206s
2022-11-23T02:08:16.1683878Z 
2022-11-23T02:08:16.1683956Z OK
2022-11-23T02:08:16.1684089Z 
2022-11-23T02:08:16.1684214Z Generating XML reports...
2022-11-23T02:08:16.1684827Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014519.xml
2022-11-23T02:08:16.1685540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1685976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1686551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1687023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1687251Z 
2022-11-23T02:08:16.1687343Z Running tests...
2022-11-23T02:08:16.1687750Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1688277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1688843Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1689367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21877
2022-11-23T02:08:16.1689818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21878
2022-11-23T02:08:16.1690420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1690905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1691738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1692230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1692812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1693244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1693811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1694277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1694726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1695203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1695857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1696550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1697060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1697537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1698035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy88rdtr2
2022-11-23T02:08:16.1698576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy88rdtr2/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1699090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ixaja73
2022-11-23T02:08:16.1699626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ixaja73/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1700525Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1701074Z   warnings.warn(
2022-11-23T02:08:16.1701815Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1702454Z   warnings.warn(
2022-11-23T02:08:16.1702831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1703369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1703834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1704317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1704664Z ok (6.111s)
2022-11-23T02:08:16.1704812Z 
2022-11-23T02:08:16.1705070Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1705400Z Ran 1 test in 6.111s
2022-11-23T02:08:16.1705562Z 
2022-11-23T02:08:16.1705656Z OK
2022-11-23T02:08:16.1705793Z 
2022-11-23T02:08:16.1705917Z Generating XML reports...
2022-11-23T02:08:16.1706519Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014528.xml
2022-11-23T02:08:16.1707239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1707690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1708243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1708783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1709026Z 
2022-11-23T02:08:16.1709135Z Running tests...
2022-11-23T02:08:16.1709543Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1710054Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1710643Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1711208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21996
2022-11-23T02:08:16.1711639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21997
2022-11-23T02:08:16.1712251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1712701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1713276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1713730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1714306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1714754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1715307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1715772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1716224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1716721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1717367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1718055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1718574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1719108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1719590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc02u_4d
2022-11-23T02:08:16.1720128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc02u_4d/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1720657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4vkt2i7
2022-11-23T02:08:16.1721190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4vkt2i7/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1722084Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1722634Z   warnings.warn(
2022-11-23T02:08:16.1723379Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.1723924Z   warnings.warn(
2022-11-23T02:08:16.1724280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1724770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1725250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1725764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1726126Z ok (5.759s)
2022-11-23T02:08:16.1726273Z 
2022-11-23T02:08:16.1726544Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1726857Z Ran 1 test in 5.759s
2022-11-23T02:08:16.1727018Z 
2022-11-23T02:08:16.1727111Z OK
2022-11-23T02:08:16.1727244Z 
2022-11-23T02:08:16.1727374Z Generating XML reports...
2022-11-23T02:08:16.1727975Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014537.xml
2022-11-23T02:08:16.1728676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1729123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1729710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1730186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1730417Z 
2022-11-23T02:08:16.1730528Z Running tests...
2022-11-23T02:08:16.1730917Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1731674Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1732245Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1733322Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.649s)
2022-11-23T02:08:16.1733843Z 
2022-11-23T02:08:16.1734092Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1734419Z Ran 1 test in 1.649s
2022-11-23T02:08:16.1734584Z 
2022-11-23T02:08:16.1734694Z OK (skipped=1)
2022-11-23T02:08:16.1734846Z 
2022-11-23T02:08:16.1734954Z Generating XML reports...
2022-11-23T02:08:16.1735560Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014545.xml
2022-11-23T02:08:16.1736273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1736825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1737386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1737858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1738088Z 
2022-11-23T02:08:16.1738195Z Running tests...
2022-11-23T02:08:16.1738598Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1739115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1739663Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1740186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22149
2022-11-23T02:08:16.1740617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22150
2022-11-23T02:08:16.1741223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1741672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1742243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1742697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1743344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1743805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1744365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1744834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1745294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1745798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1746437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1747121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1747644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1748114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1748443Z ok (4.315s)
2022-11-23T02:08:16.1748588Z 
2022-11-23T02:08:16.1748855Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1749180Z Ran 1 test in 4.316s
2022-11-23T02:08:16.1749346Z 
2022-11-23T02:08:16.1749422Z OK
2022-11-23T02:08:16.1749556Z 
2022-11-23T02:08:16.1749679Z Generating XML reports...
2022-11-23T02:08:16.1750282Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014549.xml
2022-11-23T02:08:16.1750996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1751428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1751999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1752465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1752694Z 
2022-11-23T02:08:16.1752785Z Running tests...
2022-11-23T02:08:16.1753185Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1753710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1754337Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1755388Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2022-11-23T02:08:16.1755901Z 
2022-11-23T02:08:16.1756171Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1756496Z Ran 1 test in 1.628s
2022-11-23T02:08:16.1756658Z 
2022-11-23T02:08:16.1756765Z OK (skipped=1)
2022-11-23T02:08:16.1756919Z 
2022-11-23T02:08:16.1757025Z Generating XML reports...
2022-11-23T02:08:16.1757626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014556.xml
2022-11-23T02:08:16.1758342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1758794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1759354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1759825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1760055Z 
2022-11-23T02:08:16.1760163Z Running tests...
2022-11-23T02:08:16.1760593Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1761135Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1761662Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1762169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22288
2022-11-23T02:08:16.1762608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22289
2022-11-23T02:08:16.1763205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1763649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1764207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1764682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1765262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1765706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1766258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1766724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1767178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1767654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1768313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1769001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1769521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1769975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1770317Z ok (5.668s)
2022-11-23T02:08:16.1770462Z 
2022-11-23T02:08:16.1770728Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1771135Z Ran 1 test in 5.668s
2022-11-23T02:08:16.1771280Z 
2022-11-23T02:08:16.1771373Z OK
2022-11-23T02:08:16.1771741Z 
2022-11-23T02:08:16.1771873Z Generating XML reports...
2022-11-23T02:08:16.1772496Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014600.xml
2022-11-23T02:08:16.1773194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1773647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1774220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1774689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1774901Z 
2022-11-23T02:08:16.1775009Z Running tests...
2022-11-23T02:08:16.1775414Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1775945Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1776432Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T02:08:16.1776720Z 
2022-11-23T02:08:16.1776982Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1777306Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1777464Z 
2022-11-23T02:08:16.1777571Z OK (skipped=1)
2022-11-23T02:08:16.1777708Z 
2022-11-23T02:08:16.1777911Z Generating XML reports...
2022-11-23T02:08:16.1778538Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014608.xml
2022-11-23T02:08:16.1779251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1779681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1780260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1780727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1780957Z 
2022-11-23T02:08:16.1781067Z Running tests...
2022-11-23T02:08:16.1781455Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1781978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1782437Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1782899Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1783375Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22436
2022-11-23T02:08:16.1783824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22437
2022-11-23T02:08:16.1784434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1784866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1785440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1785907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1786482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1786911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1787476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1787942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1788381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1788961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1789617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1790307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1790811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1791286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1791787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyevll34
2022-11-23T02:08:16.1792328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyevll34/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1792848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxwsvota
2022-11-23T02:08:16.1793381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxwsvota/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1793761Z ok (6.053s)
2022-11-23T02:08:16.1793908Z 
2022-11-23T02:08:16.1794163Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1794493Z Ran 1 test in 6.053s
2022-11-23T02:08:16.1794658Z 
2022-11-23T02:08:16.1794752Z OK
2022-11-23T02:08:16.1794887Z 
2022-11-23T02:08:16.1794992Z Generating XML reports...
2022-11-23T02:08:16.1795653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014611.xml
2022-11-23T02:08:16.1796384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1796838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1797402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1797871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1798097Z 
2022-11-23T02:08:16.1798206Z Running tests...
2022-11-23T02:08:16.1798609Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1799118Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1799594Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1800097Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1800564Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22555
2022-11-23T02:08:16.1801007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22556
2022-11-23T02:08:16.1801615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1802072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1802629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1803097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1803729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1804159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1804730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1805194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1805643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1806189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1806848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1807540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1808068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1808530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1809023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe6cxclg
2022-11-23T02:08:16.1809559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe6cxclg/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1810071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9z2h3dm
2022-11-23T02:08:16.1810608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9z2h3dm/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1811123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1811847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1812192Z ok (6.050s)
2022-11-23T02:08:16.1812339Z 
2022-11-23T02:08:16.1812617Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1813022Z Ran 1 test in 6.051s
2022-11-23T02:08:16.1813197Z 
2022-11-23T02:08:16.1813274Z OK
2022-11-23T02:08:16.1813409Z 
2022-11-23T02:08:16.1813536Z Generating XML reports...
2022-11-23T02:08:16.1814145Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014619.xml
2022-11-23T02:08:16.1814863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1815300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1815875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1816347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1816576Z 
2022-11-23T02:08:16.1816687Z Running tests...
2022-11-23T02:08:16.1817074Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1817607Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1818102Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1818617Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1819121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22674
2022-11-23T02:08:16.1819572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22675
2022-11-23T02:08:16.1820177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1820611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1821186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1821661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1822221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1822664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1823231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1823777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1824218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1824885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1825421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1826059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1826583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1827052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1827551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8egd6t2h
2022-11-23T02:08:16.1828075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8egd6t2h/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1828605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqc0c_42
2022-11-23T02:08:16.1829133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqc0c_42/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1829646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1830194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.1830562Z ok (6.157s)
2022-11-23T02:08:16.1830714Z 
2022-11-23T02:08:16.1830984Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1831295Z Ran 1 test in 6.157s
2022-11-23T02:08:16.1831456Z 
2022-11-23T02:08:16.1831551Z OK
2022-11-23T02:08:16.1831685Z 
2022-11-23T02:08:16.1831808Z Generating XML reports...
2022-11-23T02:08:16.1832417Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014628.xml
2022-11-23T02:08:16.1833120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1833572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1834143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1834600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1834832Z 
2022-11-23T02:08:16.1834941Z Running tests...
2022-11-23T02:08:16.1835341Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1835872Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1836332Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:08:16.1836831Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1837311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22793
2022-11-23T02:08:16.1837740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22794
2022-11-23T02:08:16.1838347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1838797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1839370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1839826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1840404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1840923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1841478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1841942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1842396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1842893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1843534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1844225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1844747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1845285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1845778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g6qtn1d
2022-11-23T02:08:16.1846321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g6qtn1d/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1846847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv19h31uo
2022-11-23T02:08:16.1847364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv19h31uo/_remote_module_non_scriptable.py
2022-11-23T02:08:16.1847792Z ok (6.047s)
2022-11-23T02:08:16.1847951Z 
2022-11-23T02:08:16.1848224Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1848556Z Ran 1 test in 6.047s
2022-11-23T02:08:16.1848699Z 
2022-11-23T02:08:16.1848793Z OK
2022-11-23T02:08:16.1848928Z 
2022-11-23T02:08:16.1849050Z Generating XML reports...
2022-11-23T02:08:16.1849658Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014637.xml
2022-11-23T02:08:16.1850360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1850811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1851382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1852090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1852324Z 
2022-11-23T02:08:16.1852420Z Running tests...
2022-11-23T02:08:16.1852831Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1853362Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1853848Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.1854137Z 
2022-11-23T02:08:16.1854403Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1854728Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1854886Z 
2022-11-23T02:08:16.1854993Z OK (skipped=1)
2022-11-23T02:08:16.1855149Z 
2022-11-23T02:08:16.1855255Z Generating XML reports...
2022-11-23T02:08:16.1855860Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014645.xml
2022-11-23T02:08:16.1856575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1857023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1857579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1858046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1858274Z 
2022-11-23T02:08:16.1858474Z Running tests...
2022-11-23T02:08:16.1858868Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1859398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1859938Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T02:08:16.1860258Z 
2022-11-23T02:08:16.1860520Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1860829Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1860989Z 
2022-11-23T02:08:16.1861100Z OK (skipped=1)
2022-11-23T02:08:16.1861254Z 
2022-11-23T02:08:16.1861378Z Generating XML reports...
2022-11-23T02:08:16.1861964Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014648.xml
2022-11-23T02:08:16.1862676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1863128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1863701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1864156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1864386Z 
2022-11-23T02:08:16.1864495Z Running tests...
2022-11-23T02:08:16.1864897Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1865474Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1866034Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T02:08:16.1866358Z 
2022-11-23T02:08:16.1866621Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1866949Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1867115Z 
2022-11-23T02:08:16.1867205Z OK (skipped=1)
2022-11-23T02:08:16.1867360Z 
2022-11-23T02:08:16.1867485Z Generating XML reports...
2022-11-23T02:08:16.1868082Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014650.xml
2022-11-23T02:08:16.1868789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1869225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1869799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1870269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1870499Z 
2022-11-23T02:08:16.1870591Z Running tests...
2022-11-23T02:08:16.1870992Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1871517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1872059Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T02:08:16.1872373Z 
2022-11-23T02:08:16.1872620Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1872941Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1873103Z 
2022-11-23T02:08:16.1873211Z OK (skipped=1)
2022-11-23T02:08:16.1873364Z 
2022-11-23T02:08:16.1873470Z Generating XML reports...
2022-11-23T02:08:16.1874075Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014652.xml
2022-11-23T02:08:16.1874782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1875234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1875789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1876325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1876556Z 
2022-11-23T02:08:16.1876664Z Running tests...
2022-11-23T02:08:16.1877070Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1877580Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1878125Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-11-23T02:08:16.1878443Z 
2022-11-23T02:08:16.1878705Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1879010Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1879169Z 
2022-11-23T02:08:16.1879274Z OK (skipped=1)
2022-11-23T02:08:16.1879293Z 
2022-11-23T02:08:16.1879415Z Generating XML reports...
2022-11-23T02:08:16.1879865Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014655.xml
2022-11-23T02:08:16.1880235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1880409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1880789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1880963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1881049Z 
2022-11-23T02:08:16.1881148Z Running tests...
2022-11-23T02:08:16.1881419Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1881734Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1882029Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-11-23T02:08:16.1882052Z 
2022-11-23T02:08:16.1882314Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1882426Z Ran 1 test in 0.003s
2022-11-23T02:08:16.1882445Z 
2022-11-23T02:08:16.1882553Z OK (skipped=1)
2022-11-23T02:08:16.1882571Z 
2022-11-23T02:08:16.1882693Z Generating XML reports...
2022-11-23T02:08:16.1883123Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014657.xml
2022-11-23T02:08:16.1883492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1883664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1884041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1884232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1884254Z 
2022-11-23T02:08:16.1884361Z Running tests...
2022-11-23T02:08:16.1884621Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1884934Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1885184Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.1885220Z 
2022-11-23T02:08:16.1885465Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1885576Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1885599Z 
2022-11-23T02:08:16.1885708Z OK (skipped=1)
2022-11-23T02:08:16.1885727Z 
2022-11-23T02:08:16.1885850Z Generating XML reports...
2022-11-23T02:08:16.1886295Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014659.xml
2022-11-23T02:08:16.1886663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1886910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1887294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1887466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1887504Z 
2022-11-23T02:08:16.1887595Z Running tests...
2022-11-23T02:08:16.1887856Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1888169Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1888423Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1888640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23143
2022-11-23T02:08:16.1888852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23144
2022-11-23T02:08:16.1889223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1889396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1889756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1889944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1890355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1890537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1890918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1891109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1891359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1891845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1892243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1892643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1892876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1893105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1893441Z STAGE:2022-11-23 01:47:07 23143:23143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1893772Z STAGE:2022-11-23 01:47:07 23144:23144 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1894113Z STAGE:2022-11-23 01:47:08 23143:23143 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1894447Z STAGE:2022-11-23 01:47:08 23144:23144 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1894794Z STAGE:2022-11-23 01:47:08 23143:23143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1895122Z STAGE:2022-11-23 01:47:08 23144:23144 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1895456Z STAGE:2022-11-23 01:47:08 23143:23143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1895784Z STAGE:2022-11-23 01:47:08 23144:23144 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1896116Z STAGE:2022-11-23 01:47:08 23143:23143 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1896462Z STAGE:2022-11-23 01:47:08 23143:23143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1896885Z STAGE:2022-11-23 01:47:08 23144:23144 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1897232Z STAGE:2022-11-23 01:47:08 23144:23144 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1897337Z ok (6.434s)
2022-11-23T02:08:16.1897357Z 
2022-11-23T02:08:16.1897623Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1897717Z Ran 1 test in 6.434s
2022-11-23T02:08:16.1897736Z 
2022-11-23T02:08:16.1897829Z OK
2022-11-23T02:08:16.1897848Z 
2022-11-23T02:08:16.1897975Z Generating XML reports...
2022-11-23T02:08:16.1898426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014702.xml
2022-11-23T02:08:16.1898795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1898976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1899359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1899553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1899572Z 
2022-11-23T02:08:16.1899682Z Running tests...
2022-11-23T02:08:16.1899928Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1900239Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1900565Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1900797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23262
2022-11-23T02:08:16.1901016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23263
2022-11-23T02:08:16.1901389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1901571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1901951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1902124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1902550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1902747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1903258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1903542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1903827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1904109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1904564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1904998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1905211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1905480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1905853Z STAGE:2022-11-23 01:47:16 23262:23262 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1906264Z STAGE:2022-11-23 01:47:16 23263:23263 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1906666Z STAGE:2022-11-23 01:47:17 23262:23262 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1907117Z STAGE:2022-11-23 01:47:17 23263:23263 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1907503Z STAGE:2022-11-23 01:47:17 23262:23262 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1907886Z STAGE:2022-11-23 01:47:17 23263:23263 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1908250Z STAGE:2022-11-23 01:47:17 23262:23262 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1908564Z STAGE:2022-11-23 01:47:17 23263:23263 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1937524Z STAGE:2022-11-23 01:47:17 23262:23262 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1937987Z STAGE:2022-11-23 01:47:17 23262:23262 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1938339Z STAGE:2022-11-23 01:47:17 23263:23263 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1938698Z STAGE:2022-11-23 01:47:17 23263:23263 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1938803Z ok (6.554s)
2022-11-23T02:08:16.1938826Z 
2022-11-23T02:08:16.1939096Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1939194Z Ran 1 test in 6.555s
2022-11-23T02:08:16.1939214Z 
2022-11-23T02:08:16.1939308Z OK
2022-11-23T02:08:16.1939327Z 
2022-11-23T02:08:16.1939452Z Generating XML reports...
2022-11-23T02:08:16.1940044Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014711.xml
2022-11-23T02:08:16.1940446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1940626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1941009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1941209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1941229Z 
2022-11-23T02:08:16.1941339Z Running tests...
2022-11-23T02:08:16.1941587Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1941904Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1942180Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.1942199Z 
2022-11-23T02:08:16.1942461Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1942575Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1942595Z 
2022-11-23T02:08:16.1942702Z OK (skipped=1)
2022-11-23T02:08:16.1942721Z 
2022-11-23T02:08:16.1942845Z Generating XML reports...
2022-11-23T02:08:16.1943300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014720.xml
2022-11-23T02:08:16.1943651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1943830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1944211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1944404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1944427Z 
2022-11-23T02:08:16.1944537Z Running tests...
2022-11-23T02:08:16.1944801Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1945116Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1945380Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.1945478Z 
2022-11-23T02:08:16.1945747Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1945842Z Ran 1 test in 0.002s
2022-11-23T02:08:16.1945861Z 
2022-11-23T02:08:16.1945972Z OK (skipped=1)
2022-11-23T02:08:16.1945991Z 
2022-11-23T02:08:16.1946117Z Generating XML reports...
2022-11-23T02:08:16.1946571Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014722.xml
2022-11-23T02:08:16.1946947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1947125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1947507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1947699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1947722Z 
2022-11-23T02:08:16.1947832Z Running tests...
2022-11-23T02:08:16.1948077Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1948392Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1948671Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1948893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23447
2022-11-23T02:08:16.1949163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23448
2022-11-23T02:08:16.1949547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1949721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1950089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1950266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1950628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1950799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1951173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1951365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1951617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1951863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1952264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1952665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1952878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1953108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1953443Z STAGE:2022-11-23 01:47:29 23447:23447 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1953772Z STAGE:2022-11-23 01:47:29 23448:23448 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1954109Z STAGE:2022-11-23 01:47:30 23447:23447 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1954443Z STAGE:2022-11-23 01:47:30 23448:23448 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1954789Z STAGE:2022-11-23 01:47:30 23447:23447 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1955206Z STAGE:2022-11-23 01:47:30 23448:23448 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1955315Z ok (6.554s)
2022-11-23T02:08:16.1955335Z 
2022-11-23T02:08:16.1955582Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1955698Z Ran 1 test in 6.554s
2022-11-23T02:08:16.1955717Z 
2022-11-23T02:08:16.1955809Z OK
2022-11-23T02:08:16.1955828Z 
2022-11-23T02:08:16.1955953Z Generating XML reports...
2022-11-23T02:08:16.1956413Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014725.xml
2022-11-23T02:08:16.1956784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1956960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1957341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1957520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1957557Z 
2022-11-23T02:08:16.1957649Z Running tests...
2022-11-23T02:08:16.1957912Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1958227Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1958505Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1958775Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23566
2022-11-23T02:08:16.1959005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23567
2022-11-23T02:08:16.1959377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1959554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1959923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1960115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1960479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1960654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1961029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1961219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1961465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1961710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1962095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1962494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1962725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1962955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1963293Z STAGE:2022-11-23 01:47:39 23566:23566 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1963616Z STAGE:2022-11-23 01:47:39 23567:23567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.1963950Z STAGE:2022-11-23 01:47:39 23566:23566 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1964276Z STAGE:2022-11-23 01:47:39 23567:23567 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.1964681Z STAGE:2022-11-23 01:47:39 23566:23566 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1965010Z STAGE:2022-11-23 01:47:39 23567:23567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.1965119Z ok (6.567s)
2022-11-23T02:08:16.1965138Z 
2022-11-23T02:08:16.1965402Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1965517Z Ran 1 test in 6.567s
2022-11-23T02:08:16.1965536Z 
2022-11-23T02:08:16.1965630Z OK
2022-11-23T02:08:16.1965649Z 
2022-11-23T02:08:16.1965780Z Generating XML reports...
2022-11-23T02:08:16.1966230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014734.xml
2022-11-23T02:08:16.1966600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1966777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1967141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1967334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1967353Z 
2022-11-23T02:08:16.1967462Z Running tests...
2022-11-23T02:08:16.1967723Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1968038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1968362Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1969132Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2022-11-23T02:08:16.1969155Z 
2022-11-23T02:08:16.1969420Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1969534Z Ran 1 test in 1.636s
2022-11-23T02:08:16.1969553Z 
2022-11-23T02:08:16.1969643Z OK (skipped=1)
2022-11-23T02:08:16.1969680Z 
2022-11-23T02:08:16.1969788Z Generating XML reports...
2022-11-23T02:08:16.1970234Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014743.xml
2022-11-23T02:08:16.1970607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1970782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1971160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1971351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1971374Z 
2022-11-23T02:08:16.1971739Z Running tests...
2022-11-23T02:08:16.1972023Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1972324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1972599Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1973346Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.616s)
2022-11-23T02:08:16.1973367Z 
2022-11-23T02:08:16.1973626Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1973738Z Ran 1 test in 1.617s
2022-11-23T02:08:16.1973757Z 
2022-11-23T02:08:16.1973867Z OK (skipped=1)
2022-11-23T02:08:16.1973968Z 
2022-11-23T02:08:16.1974101Z Generating XML reports...
2022-11-23T02:08:16.1974556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014747.xml
2022-11-23T02:08:16.1974927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1975102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1975468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1975663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1975683Z 
2022-11-23T02:08:16.1975792Z Running tests...
2022-11-23T02:08:16.1976055Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1976369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1976649Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1976869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23753
2022-11-23T02:08:16.1977088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23754
2022-11-23T02:08:16.1977440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1977619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1978049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1978237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1978616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1978809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1979190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1979380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1979626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1979856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1980264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1980661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1980893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1981124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1981227Z ok (5.553s)
2022-11-23T02:08:16.1981246Z 
2022-11-23T02:08:16.1981513Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1981626Z Ran 1 test in 5.554s
2022-11-23T02:08:16.1981645Z 
2022-11-23T02:08:16.1981721Z OK
2022-11-23T02:08:16.1981756Z 
2022-11-23T02:08:16.1981864Z Generating XML reports...
2022-11-23T02:08:16.1982314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014751.xml
2022-11-23T02:08:16.1982686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1982860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1983239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1983490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1983509Z 
2022-11-23T02:08:16.1983620Z Running tests...
2022-11-23T02:08:16.1983883Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1984182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1984458Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1984680Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23868
2022-11-23T02:08:16.1984901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23869
2022-11-23T02:08:16.1985272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1985449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1985827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1986021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1986365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1986543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1986919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1987163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1987418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1987667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1988070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1988468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.1988700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.1988910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.1989156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.1989400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.1989797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.1990185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.1990430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:08:16.1990669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:08:16.1991069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.1991458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.1991706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T02:08:16.1991926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T02:08:16.1992316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:08:16.1992704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:08:16.1992863Z ok (6.550s)
2022-11-23T02:08:16.1992883Z 
2022-11-23T02:08:16.1993151Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1993263Z Ran 1 test in 6.550s
2022-11-23T02:08:16.1993282Z 
2022-11-23T02:08:16.1993375Z OK
2022-11-23T02:08:16.1993394Z 
2022-11-23T02:08:16.1993519Z Generating XML reports...
2022-11-23T02:08:16.1993954Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014759.xml
2022-11-23T02:08:16.1994332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1994512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1994894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1995090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1995109Z 
2022-11-23T02:08:16.1995217Z Running tests...
2022-11-23T02:08:16.1995480Z ----------------------------------------------------------------------
2022-11-23T02:08:16.1995796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.1996054Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.1996256Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24019
2022-11-23T02:08:16.1996518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24020
2022-11-23T02:08:16.1996899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1997076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1997456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1997650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1998013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.1998186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.1998542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.1998735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.1998979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.1999224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.1999624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2000021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2000251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2000481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2000583Z ok (5.762s)
2022-11-23T02:08:16.2000603Z 
2022-11-23T02:08:16.2000854Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2000966Z Ran 1 test in 5.762s
2022-11-23T02:08:16.2000986Z 
2022-11-23T02:08:16.2001080Z OK
2022-11-23T02:08:16.2001099Z 
2022-11-23T02:08:16.2001224Z Generating XML reports...
2022-11-23T02:08:16.2001676Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014808.xml
2022-11-23T02:08:16.2002105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2002281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2002658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2002850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2002869Z 
2022-11-23T02:08:16.2002962Z Running tests...
2022-11-23T02:08:16.2003230Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2003599Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2004026Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2004045Z 
2022-11-23T02:08:16.2004306Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2004421Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2004440Z 
2022-11-23T02:08:16.2004548Z OK (skipped=1)
2022-11-23T02:08:16.2004567Z 
2022-11-23T02:08:16.2004692Z Generating XML reports...
2022-11-23T02:08:16.2005122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014817.xml
2022-11-23T02:08:16.2005496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2005728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2006119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2006313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2006332Z 
2022-11-23T02:08:16.2006442Z Running tests...
2022-11-23T02:08:16.2006706Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2007023Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2007442Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2007461Z 
2022-11-23T02:08:16.2007703Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2007813Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2007832Z 
2022-11-23T02:08:16.2007935Z OK (skipped=1)
2022-11-23T02:08:16.2007954Z 
2022-11-23T02:08:16.2008081Z Generating XML reports...
2022-11-23T02:08:16.2008525Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014819.xml
2022-11-23T02:08:16.2008895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2009071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2009456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2009649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2009668Z 
2022-11-23T02:08:16.2009758Z Running tests...
2022-11-23T02:08:16.2010022Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2010336Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2010767Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2010787Z 
2022-11-23T02:08:16.2011047Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2011158Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2011177Z 
2022-11-23T02:08:16.2011284Z OK (skipped=1)
2022-11-23T02:08:16.2011354Z 
2022-11-23T02:08:16.2011752Z Generating XML reports...
2022-11-23T02:08:16.2012221Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014822.xml
2022-11-23T02:08:16.2012574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2012752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2013134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2013329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2013349Z 
2022-11-23T02:08:16.2013458Z Running tests...
2022-11-23T02:08:16.2013722Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2014035Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2014458Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2014478Z 
2022-11-23T02:08:16.2014735Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2014829Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2014848Z 
2022-11-23T02:08:16.2014957Z OK (skipped=1)
2022-11-23T02:08:16.2014976Z 
2022-11-23T02:08:16.2015098Z Generating XML reports...
2022-11-23T02:08:16.2015623Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014824.xml
2022-11-23T02:08:16.2016014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2016192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2016572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2016769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2016788Z 
2022-11-23T02:08:16.2016880Z Running tests...
2022-11-23T02:08:16.2017141Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2017453Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2017864Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2017883Z 
2022-11-23T02:08:16.2018146Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2018259Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2018278Z 
2022-11-23T02:08:16.2018386Z OK (skipped=1)
2022-11-23T02:08:16.2018404Z 
2022-11-23T02:08:16.2018526Z Generating XML reports...
2022-11-23T02:08:16.2018974Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014826.xml
2022-11-23T02:08:16.2019332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2019508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2019886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2020080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2020100Z 
2022-11-23T02:08:16.2020207Z Running tests...
2022-11-23T02:08:16.2020474Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2020789Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2021198Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2021286Z 
2022-11-23T02:08:16.2021555Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2021651Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2021670Z 
2022-11-23T02:08:16.2021779Z OK (skipped=1)
2022-11-23T02:08:16.2021798Z 
2022-11-23T02:08:16.2021922Z Generating XML reports...
2022-11-23T02:08:16.2022371Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014829.xml
2022-11-23T02:08:16.2022743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2022923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2023303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2023494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2023513Z 
2022-11-23T02:08:16.2023626Z Running tests...
2022-11-23T02:08:16.2023872Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2024185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2024602Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2024622Z 
2022-11-23T02:08:16.2024881Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2024992Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2025011Z 
2022-11-23T02:08:16.2025166Z OK (skipped=1)
2022-11-23T02:08:16.2025186Z 
2022-11-23T02:08:16.2025316Z Generating XML reports...
2022-11-23T02:08:16.2025767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014831.xml
2022-11-23T02:08:16.2026140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2026304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2026682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2026874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2026894Z 
2022-11-23T02:08:16.2027002Z Running tests...
2022-11-23T02:08:16.2027264Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2027579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2027984Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2028004Z 
2022-11-23T02:08:16.2028273Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2028368Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2028407Z 
2022-11-23T02:08:16.2028497Z OK (skipped=1)
2022-11-23T02:08:16.2028515Z 
2022-11-23T02:08:16.2028640Z Generating XML reports...
2022-11-23T02:08:16.2029089Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014834.xml
2022-11-23T02:08:16.2029460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2029637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2030017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2030207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2030227Z 
2022-11-23T02:08:16.2030336Z Running tests...
2022-11-23T02:08:16.2030581Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2030894Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2031386Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2031406Z 
2022-11-23T02:08:16.2031667Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2031780Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2031799Z 
2022-11-23T02:08:16.2031908Z OK (skipped=1)
2022-11-23T02:08:16.2031926Z 
2022-11-23T02:08:16.2032054Z Generating XML reports...
2022-11-23T02:08:16.2032502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014836.xml
2022-11-23T02:08:16.2032871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2033028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2033406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2033602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2033621Z 
2022-11-23T02:08:16.2033730Z Running tests...
2022-11-23T02:08:16.2033992Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2034304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2034669Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2034691Z 
2022-11-23T02:08:16.2034955Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2035068Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2035088Z 
2022-11-23T02:08:16.2035178Z OK (skipped=1)
2022-11-23T02:08:16.2035197Z 
2022-11-23T02:08:16.2035320Z Generating XML reports...
2022-11-23T02:08:16.2035770Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014838.xml
2022-11-23T02:08:16.2036137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2036312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2036691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2036887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2036909Z 
2022-11-23T02:08:16.2037018Z Running tests...
2022-11-23T02:08:16.2037263Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2037579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2037978Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2038000Z 
2022-11-23T02:08:16.2038262Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2038373Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2038392Z 
2022-11-23T02:08:16.2038501Z OK (skipped=1)
2022-11-23T02:08:16.2038520Z 
2022-11-23T02:08:16.2038645Z Generating XML reports...
2022-11-23T02:08:16.2039090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014841.xml
2022-11-23T02:08:16.2039461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2039620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2039998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2040188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2040257Z 
2022-11-23T02:08:16.2040374Z Running tests...
2022-11-23T02:08:16.2040639Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2040953Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2041360Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2041380Z 
2022-11-23T02:08:16.2041640Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2041755Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2041774Z 
2022-11-23T02:08:16.2041865Z OK (skipped=1)
2022-11-23T02:08:16.2041883Z 
2022-11-23T02:08:16.2042008Z Generating XML reports...
2022-11-23T02:08:16.2042455Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014843.xml
2022-11-23T02:08:16.2042828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2043007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2043386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2043578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2043597Z 
2022-11-23T02:08:16.2043706Z Running tests...
2022-11-23T02:08:16.2043965Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2044314Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2044720Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.2044739Z 
2022-11-23T02:08:16.2045001Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2045116Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2045134Z 
2022-11-23T02:08:16.2045242Z OK (skipped=1)
2022-11-23T02:08:16.2045261Z 
2022-11-23T02:08:16.2045384Z Generating XML reports...
2022-11-23T02:08:16.2045829Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014846.xml
2022-11-23T02:08:16.2046199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2046373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2046735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2046928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2046946Z 
2022-11-23T02:08:16.2047055Z Running tests...
2022-11-23T02:08:16.2047315Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2047630Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2047921Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2047940Z 
2022-11-23T02:08:16.2048201Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2048312Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2048331Z 
2022-11-23T02:08:16.2048421Z OK (skipped=1)
2022-11-23T02:08:16.2048458Z 
2022-11-23T02:08:16.2048564Z Generating XML reports...
2022-11-23T02:08:16.2049010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014848.xml
2022-11-23T02:08:16.2049382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2049556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2049996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2050188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2050208Z 
2022-11-23T02:08:16.2050316Z Running tests...
2022-11-23T02:08:16.2050576Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2050868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2051145Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2051165Z 
2022-11-23T02:08:16.2051621Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2051746Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2051766Z 
2022-11-23T02:08:16.2051874Z OK (skipped=1)
2022-11-23T02:08:16.2051893Z 
2022-11-23T02:08:16.2052019Z Generating XML reports...
2022-11-23T02:08:16.2052475Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014850.xml
2022-11-23T02:08:16.2052846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2053022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2053385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2053655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2053679Z 
2022-11-23T02:08:16.2053796Z Running tests...
2022-11-23T02:08:16.2054062Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2054377Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2054654Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2054677Z 
2022-11-23T02:08:16.2054938Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2055050Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2055069Z 
2022-11-23T02:08:16.2055159Z OK (skipped=1)
2022-11-23T02:08:16.2055195Z 
2022-11-23T02:08:16.2055302Z Generating XML reports...
2022-11-23T02:08:16.2055753Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014853.xml
2022-11-23T02:08:16.2056127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2056303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2056682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2056876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2056899Z 
2022-11-23T02:08:16.2057010Z Running tests...
2022-11-23T02:08:16.2057274Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2057571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2057854Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2057873Z 
2022-11-23T02:08:16.2058129Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2058245Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2058264Z 
2022-11-23T02:08:16.2058371Z OK (skipped=1)
2022-11-23T02:08:16.2058390Z 
2022-11-23T02:08:16.2058514Z Generating XML reports...
2022-11-23T02:08:16.2058960Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014855.xml
2022-11-23T02:08:16.2059327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2059576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2059945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2060138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2060157Z 
2022-11-23T02:08:16.2060266Z Running tests...
2022-11-23T02:08:16.2060528Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2060846Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2061123Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2061142Z 
2022-11-23T02:08:16.2061407Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2061523Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2061542Z 
2022-11-23T02:08:16.2061650Z OK (skipped=1)
2022-11-23T02:08:16.2061669Z 
2022-11-23T02:08:16.2061775Z Generating XML reports...
2022-11-23T02:08:16.2062222Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014858.xml
2022-11-23T02:08:16.2062589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2062764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2063191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2063391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2063410Z 
2022-11-23T02:08:16.2063520Z Running tests...
2022-11-23T02:08:16.2063784Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2064081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2064352Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2064371Z 
2022-11-23T02:08:16.2064634Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2064743Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2064762Z 
2022-11-23T02:08:16.2064870Z OK (skipped=1)
2022-11-23T02:08:16.2064889Z 
2022-11-23T02:08:16.2065013Z Generating XML reports...
2022-11-23T02:08:16.2065457Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014900.xml
2022-11-23T02:08:16.2065824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2066001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2066364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2066556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2066576Z 
2022-11-23T02:08:16.2066686Z Running tests...
2022-11-23T02:08:16.2066948Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2067257Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2067530Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2067550Z 
2022-11-23T02:08:16.2067809Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2067920Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2067939Z 
2022-11-23T02:08:16.2068047Z OK (skipped=1)
2022-11-23T02:08:16.2068065Z 
2022-11-23T02:08:16.2068171Z Generating XML reports...
2022-11-23T02:08:16.2068669Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014902.xml
2022-11-23T02:08:16.2069035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2069209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2069586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2069778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2069800Z 
2022-11-23T02:08:16.2069909Z Running tests...
2022-11-23T02:08:16.2070171Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2070465Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2070740Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2070763Z 
2022-11-23T02:08:16.2071024Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2071136Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2071155Z 
2022-11-23T02:08:16.2071263Z OK (skipped=1)
2022-11-23T02:08:16.2071281Z 
2022-11-23T02:08:16.2071403Z Generating XML reports...
2022-11-23T02:08:16.2071849Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014905.xml
2022-11-23T02:08:16.2072260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2072444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2072806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2072999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2073022Z 
2022-11-23T02:08:16.2073133Z Running tests...
2022-11-23T02:08:16.2073396Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2073708Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2073975Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2073994Z 
2022-11-23T02:08:16.2074253Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2074365Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2074387Z 
2022-11-23T02:08:16.2074496Z OK (skipped=1)
2022-11-23T02:08:16.2074515Z 
2022-11-23T02:08:16.2074620Z Generating XML reports...
2022-11-23T02:08:16.2075064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014907.xml
2022-11-23T02:08:16.2075434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2075616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2075992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2076183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2076202Z 
2022-11-23T02:08:16.2076310Z Running tests...
2022-11-23T02:08:16.2076578Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2076894Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2077136Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2077156Z 
2022-11-23T02:08:16.2077418Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2077529Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2077593Z 
2022-11-23T02:08:16.2077707Z OK (skipped=1)
2022-11-23T02:08:16.2077726Z 
2022-11-23T02:08:16.2077850Z Generating XML reports...
2022-11-23T02:08:16.2078297Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014909.xml
2022-11-23T02:08:16.2078667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2078843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2079225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2079401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2079420Z 
2022-11-23T02:08:16.2079530Z Running tests...
2022-11-23T02:08:16.2079795Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2080107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2080372Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2080391Z 
2022-11-23T02:08:16.2080652Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2080765Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2080783Z 
2022-11-23T02:08:16.2080891Z OK (skipped=1)
2022-11-23T02:08:16.2080910Z 
2022-11-23T02:08:16.2081015Z Generating XML reports...
2022-11-23T02:08:16.2081506Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014912.xml
2022-11-23T02:08:16.2081890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2082067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2082446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2082642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2082661Z 
2022-11-23T02:08:16.2082769Z Running tests...
2022-11-23T02:08:16.2083028Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2083340Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2083601Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-11-23T02:08:16.2083624Z 
2022-11-23T02:08:16.2083882Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2083994Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2084013Z 
2022-11-23T02:08:16.2084121Z OK (skipped=1)
2022-11-23T02:08:16.2084140Z 
2022-11-23T02:08:16.2084263Z Generating XML reports...
2022-11-23T02:08:16.2084709Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014914.xml
2022-11-23T02:08:16.2085083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2085260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2085636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2085810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2085829Z 
2022-11-23T02:08:16.2085941Z Running tests...
2022-11-23T02:08:16.2086202Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2086513Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2086803Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-11-23T02:08:16.2086871Z 
2022-11-23T02:08:16.2087136Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2087249Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2087268Z 
2022-11-23T02:08:16.2087377Z OK (skipped=1)
2022-11-23T02:08:16.2087397Z 
2022-11-23T02:08:16.2087519Z Generating XML reports...
2022-11-23T02:08:16.2087945Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014917.xml
2022-11-23T02:08:16.2088319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2088495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2088872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2089061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2089083Z 
2022-11-23T02:08:16.2089195Z Running tests...
2022-11-23T02:08:16.2089454Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2089766Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2090015Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2090054Z 
2022-11-23T02:08:16.2090297Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2090407Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2090426Z 
2022-11-23T02:08:16.2090580Z OK (skipped=1)
2022-11-23T02:08:16.2090601Z 
2022-11-23T02:08:16.2090731Z Generating XML reports...
2022-11-23T02:08:16.2091173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014919.xml
2022-11-23T02:08:16.2091776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2091963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2092348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2092521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2092541Z 
2022-11-23T02:08:16.2092652Z Running tests...
2022-11-23T02:08:16.2092913Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2093227Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2093491Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2093713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25025
2022-11-23T02:08:16.2093934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25026
2022-11-23T02:08:16.2094304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2094462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2094843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2095032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2095395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2095568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2095940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2096128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2096375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2096713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2097100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2097503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2097739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2097967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2098067Z ok (5.671s)
2022-11-23T02:08:16.2098087Z 
2022-11-23T02:08:16.2098351Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2098462Z Ran 1 test in 5.671s
2022-11-23T02:08:16.2098481Z 
2022-11-23T02:08:16.2098575Z OK
2022-11-23T02:08:16.2098594Z 
2022-11-23T02:08:16.2098715Z Generating XML reports...
2022-11-23T02:08:16.2099150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014921.xml
2022-11-23T02:08:16.2099519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2099695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2100146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2100340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2100360Z 
2022-11-23T02:08:16.2100470Z Running tests...
2022-11-23T02:08:16.2100730Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2101045Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2101292Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2101323Z 
2022-11-23T02:08:16.2101564Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2101669Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2101688Z 
2022-11-23T02:08:16.2101791Z OK (skipped=1)
2022-11-23T02:08:16.2101810Z 
2022-11-23T02:08:16.2101931Z Generating XML reports...
2022-11-23T02:08:16.2102378Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014930.xml
2022-11-23T02:08:16.2102743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2102910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2103288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2103509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2103529Z 
2022-11-23T02:08:16.2103640Z Running tests...
2022-11-23T02:08:16.2103904Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2104214Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2104480Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2104500Z 
2022-11-23T02:08:16.2104757Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2104864Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2104883Z 
2022-11-23T02:08:16.2104990Z OK (skipped=1)
2022-11-23T02:08:16.2105009Z 
2022-11-23T02:08:16.2105128Z Generating XML reports...
2022-11-23T02:08:16.2105554Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014932.xml
2022-11-23T02:08:16.2105982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2106156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2106531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2106719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2106738Z 
2022-11-23T02:08:16.2106844Z Running tests...
2022-11-23T02:08:16.2107098Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2107407Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2107661Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2107696Z 
2022-11-23T02:08:16.2107938Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2108047Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2108067Z 
2022-11-23T02:08:16.2108171Z OK (skipped=1)
2022-11-23T02:08:16.2108190Z 
2022-11-23T02:08:16.2108310Z Generating XML reports...
2022-11-23T02:08:16.2108753Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014934.xml
2022-11-23T02:08:16.2109118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2109335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2109713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2109887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2109922Z 
2022-11-23T02:08:16.2110014Z Running tests...
2022-11-23T02:08:16.2110273Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2110575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2110826Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2111043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25239
2022-11-23T02:08:16.2111249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25240
2022-11-23T02:08:16.2111612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2111776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2112133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2112314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2112670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2112830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2113190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2113370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2113606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2113844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2114230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2114618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2114891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2115110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2115439Z STAGE:2022-11-23 01:49:42 25240:25240 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2115754Z STAGE:2022-11-23 01:49:42 25239:25239 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2116084Z STAGE:2022-11-23 01:49:42 25240:25240 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2116407Z STAGE:2022-11-23 01:49:42 25239:25239 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2116743Z STAGE:2022-11-23 01:49:42 25240:25240 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2117071Z STAGE:2022-11-23 01:49:42 25239:25239 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2117395Z STAGE:2022-11-23 01:49:42 25240:25240 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2117707Z STAGE:2022-11-23 01:49:42 25239:25239 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2118042Z STAGE:2022-11-23 01:49:42 25240:25240 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2118384Z STAGE:2022-11-23 01:49:42 25240:25240 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2118759Z STAGE:2022-11-23 01:49:42 25239:25239 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2119111Z STAGE:2022-11-23 01:49:42 25239:25239 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2119436Z STAGE:2022-11-23 01:49:42 25240:25240 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2119753Z STAGE:2022-11-23 01:49:42 25239:25239 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2120072Z STAGE:2022-11-23 01:49:42 25240:25240 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2120404Z STAGE:2022-11-23 01:49:42 25240:25240 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2120727Z STAGE:2022-11-23 01:49:42 25239:25239 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2121069Z STAGE:2022-11-23 01:49:42 25239:25239 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2121173Z ok (5.657s)
2022-11-23T02:08:16.2121193Z 
2022-11-23T02:08:16.2121460Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2121571Z Ran 1 test in 5.657s
2022-11-23T02:08:16.2121590Z 
2022-11-23T02:08:16.2121681Z OK
2022-11-23T02:08:16.2121701Z 
2022-11-23T02:08:16.2121825Z Generating XML reports...
2022-11-23T02:08:16.2122261Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014937.xml
2022-11-23T02:08:16.2122627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2122794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2123168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2123351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2123370Z 
2022-11-23T02:08:16.2123473Z Running tests...
2022-11-23T02:08:16.2123728Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2124032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2124284Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2124557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25357
2022-11-23T02:08:16.2124773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25358
2022-11-23T02:08:16.2125143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2125316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2125690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2125883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2126247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2126419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2126771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2126958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2127200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2127434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2127824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2128252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2128482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2128700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2129025Z STAGE:2022-11-23 01:49:50 25358:25358 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2129335Z STAGE:2022-11-23 01:49:50 25357:25357 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2129661Z STAGE:2022-11-23 01:49:50 25357:25357 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2129986Z STAGE:2022-11-23 01:49:50 25358:25358 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2130321Z STAGE:2022-11-23 01:49:50 25357:25357 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2130660Z STAGE:2022-11-23 01:49:50 25358:25358 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2130978Z STAGE:2022-11-23 01:49:50 25358:25358 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2131287Z STAGE:2022-11-23 01:49:50 25357:25357 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2131845Z STAGE:2022-11-23 01:49:50 25358:25358 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2132194Z STAGE:2022-11-23 01:49:50 25358:25358 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2132509Z STAGE:2022-11-23 01:49:50 25357:25357 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2132843Z STAGE:2022-11-23 01:49:50 25357:25357 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2133159Z STAGE:2022-11-23 01:49:50 25358:25358 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2133480Z STAGE:2022-11-23 01:49:50 25357:25357 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2133803Z STAGE:2022-11-23 01:49:50 25358:25358 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2134138Z STAGE:2022-11-23 01:49:50 25358:25358 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2134551Z STAGE:2022-11-23 01:49:50 25357:25357 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2134885Z STAGE:2022-11-23 01:49:50 25357:25357 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2134971Z ok (5.644s)
2022-11-23T02:08:16.2134999Z 
2022-11-23T02:08:16.2135246Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2135349Z Ran 1 test in 5.644s
2022-11-23T02:08:16.2135368Z 
2022-11-23T02:08:16.2135452Z OK
2022-11-23T02:08:16.2135471Z 
2022-11-23T02:08:16.2135589Z Generating XML reports...
2022-11-23T02:08:16.2136029Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014945.xml
2022-11-23T02:08:16.2136388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2136553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2136927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2137102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2137121Z 
2022-11-23T02:08:16.2137223Z Running tests...
2022-11-23T02:08:16.2137473Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2137775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2138100Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2138320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25475
2022-11-23T02:08:16.2138529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25476
2022-11-23T02:08:16.2138893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2139056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2139428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2139610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2139963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2140124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2140489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2140671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2140907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2141147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2141531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2141918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2142138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2142361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2142688Z STAGE:2022-11-23 01:49:58 25476:25476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2143000Z STAGE:2022-11-23 01:49:58 25475:25475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2143325Z STAGE:2022-11-23 01:49:58 25476:25476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2143713Z STAGE:2022-11-23 01:49:58 25475:25475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2144049Z STAGE:2022-11-23 01:49:58 25476:25476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2144376Z STAGE:2022-11-23 01:49:58 25475:25475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2144694Z STAGE:2022-11-23 01:49:58 25476:25476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2145006Z STAGE:2022-11-23 01:49:58 25475:25475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2145390Z STAGE:2022-11-23 01:49:58 25476:25476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2145730Z STAGE:2022-11-23 01:49:58 25476:25476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2146049Z STAGE:2022-11-23 01:49:58 25475:25475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2146390Z STAGE:2022-11-23 01:49:58 25475:25475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2146705Z STAGE:2022-11-23 01:49:58 25476:25476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2147019Z STAGE:2022-11-23 01:49:58 25475:25475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2147332Z STAGE:2022-11-23 01:49:58 25476:25476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2147710Z STAGE:2022-11-23 01:49:58 25476:25476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2148041Z STAGE:2022-11-23 01:49:58 25475:25475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2148377Z STAGE:2022-11-23 01:49:58 25475:25475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2148471Z ok (5.599s)
2022-11-23T02:08:16.2148494Z 
2022-11-23T02:08:16.2148750Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2148852Z Ran 1 test in 5.599s
2022-11-23T02:08:16.2148872Z 
2022-11-23T02:08:16.2148954Z OK
2022-11-23T02:08:16.2148973Z 
2022-11-23T02:08:16.2149088Z Generating XML reports...
2022-11-23T02:08:16.2149522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014953.xml
2022-11-23T02:08:16.2149883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2150052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2150424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2150607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2150627Z 
2022-11-23T02:08:16.2150728Z Running tests...
2022-11-23T02:08:16.2150980Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2151283Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2151511Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:08:16.2151538Z 
2022-11-23T02:08:16.2151781Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2151882Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2151900Z 
2022-11-23T02:08:16.2151999Z OK (skipped=1)
2022-11-23T02:08:16.2152022Z 
2022-11-23T02:08:16.2152134Z Generating XML reports...
2022-11-23T02:08:16.2152569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015001.xml
2022-11-23T02:08:16.2152927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2153148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2153515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2153690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2153709Z 
2022-11-23T02:08:16.2153810Z Running tests...
2022-11-23T02:08:16.2154061Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2154363Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2154610Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:08:16.2154629Z 
2022-11-23T02:08:16.2154884Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2154985Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2155005Z 
2022-11-23T02:08:16.2155102Z OK (skipped=1)
2022-11-23T02:08:16.2155124Z 
2022-11-23T02:08:16.2155237Z Generating XML reports...
2022-11-23T02:08:16.2155669Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015004.xml
2022-11-23T02:08:16.2156024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2156192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2156560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2156797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2156817Z 
2022-11-23T02:08:16.2156927Z Running tests...
2022-11-23T02:08:16.2157182Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2157491Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2157729Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2157941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25659
2022-11-23T02:08:16.2158156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25660
2022-11-23T02:08:16.2158514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2158684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2159057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2159249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2159603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2159772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2160131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2160323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2160565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2160799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2161202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2161595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2161818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2162043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2162189Z ok (5.753s)
2022-11-23T02:08:16.2162228Z 
2022-11-23T02:08:16.2162476Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2162588Z Ran 1 test in 5.753s
2022-11-23T02:08:16.2162607Z 
2022-11-23T02:08:16.2162696Z OK
2022-11-23T02:08:16.2162715Z 
2022-11-23T02:08:16.2162830Z Generating XML reports...
2022-11-23T02:08:16.2163270Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015006.xml
2022-11-23T02:08:16.2163632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2163800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2164166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2164341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2164363Z 
2022-11-23T02:08:16.2164461Z Running tests...
2022-11-23T02:08:16.2164712Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2165016Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2165270Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2165478Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25776
2022-11-23T02:08:16.2165748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25777
2022-11-23T02:08:16.2166129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2166287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2166655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2166845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2167196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2167368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2167730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2167915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2168152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2168396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2168778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2169171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2169397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2169616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2169714Z ok (5.645s)
2022-11-23T02:08:16.2169734Z 
2022-11-23T02:08:16.2169994Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2170100Z Ran 1 test in 5.645s
2022-11-23T02:08:16.2170120Z 
2022-11-23T02:08:16.2170209Z OK
2022-11-23T02:08:16.2170228Z 
2022-11-23T02:08:16.2170340Z Generating XML reports...
2022-11-23T02:08:16.2170777Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015015.xml
2022-11-23T02:08:16.2171142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2171366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2171986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2172172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2172193Z 
2022-11-23T02:08:16.2172299Z Running tests...
2022-11-23T02:08:16.2172553Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2172868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2173114Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:08:16.2173139Z 
2022-11-23T02:08:16.2173390Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2173494Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2173513Z 
2022-11-23T02:08:16.2173617Z OK (skipped=1)
2022-11-23T02:08:16.2173636Z 
2022-11-23T02:08:16.2173750Z Generating XML reports...
2022-11-23T02:08:16.2174195Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015023.xml
2022-11-23T02:08:16.2174553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2174727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2175210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2175396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2175415Z 
2022-11-23T02:08:16.2175521Z Running tests...
2022-11-23T02:08:16.2175775Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2176092Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2176351Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2176570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25926
2022-11-23T02:08:16.2176778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25927
2022-11-23T02:08:16.2177143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2177305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2177673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2177861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2178213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2178384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2178747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2178932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2179170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2179415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2179799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2180189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2180418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2180732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.2180952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2181191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.2181583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2181974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2182066Z ok (5.636s)
2022-11-23T02:08:16.2182086Z 
2022-11-23T02:08:16.2182331Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2182440Z Ran 1 test in 5.636s
2022-11-23T02:08:16.2182459Z 
2022-11-23T02:08:16.2182544Z OK
2022-11-23T02:08:16.2182563Z 
2022-11-23T02:08:16.2182683Z Generating XML reports...
2022-11-23T02:08:16.2183123Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015025.xml
2022-11-23T02:08:16.2183488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2183653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2184077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2184267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2184286Z 
2022-11-23T02:08:16.2184378Z Running tests...
2022-11-23T02:08:16.2184640Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2184952Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2185200Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:08:16.2185220Z 
2022-11-23T02:08:16.2185469Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2185580Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2185600Z 
2022-11-23T02:08:16.2185702Z OK (skipped=1)
2022-11-23T02:08:16.2185722Z 
2022-11-23T02:08:16.2185833Z Generating XML reports...
2022-11-23T02:08:16.2186267Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015033.xml
2022-11-23T02:08:16.2186636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2186800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2187175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2187360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2187378Z 
2022-11-23T02:08:16.2187485Z Running tests...
2022-11-23T02:08:16.2187736Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2188048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2188302Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2188506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26078
2022-11-23T02:08:16.2188724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26079
2022-11-23T02:08:16.2189091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2189255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2189688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2189868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2190232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2190404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2190758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2190944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2191187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2191419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2191818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2192206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2192435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2192660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2192809Z skip: Skipped due to small world size. (4.392s)
2022-11-23T02:08:16.2192828Z 
2022-11-23T02:08:16.2193123Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2193238Z Ran 1 test in 4.392s
2022-11-23T02:08:16.2193258Z 
2022-11-23T02:08:16.2193356Z OK (skipped=1)
2022-11-23T02:08:16.2193375Z 
2022-11-23T02:08:16.2193496Z Generating XML reports...
2022-11-23T02:08:16.2193937Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015036.xml
2022-11-23T02:08:16.2194308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2194474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2194849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2195036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2195055Z 
2022-11-23T02:08:16.2195147Z Running tests...
2022-11-23T02:08:16.2195406Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2195715Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2195993Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:08:16.2196015Z 
2022-11-23T02:08:16.2196273Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2196373Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2196392Z 
2022-11-23T02:08:16.2196499Z OK (skipped=1)
2022-11-23T02:08:16.2196517Z 
2022-11-23T02:08:16.2196635Z Generating XML reports...
2022-11-23T02:08:16.2197061Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015043.xml
2022-11-23T02:08:16.2197420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2197598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2197965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2198154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2198173Z 
2022-11-23T02:08:16.2198278Z Running tests...
2022-11-23T02:08:16.2198589Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2198896Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2199181Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:08:16.2199201Z 
2022-11-23T02:08:16.2199457Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2199552Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2199571Z 
2022-11-23T02:08:16.2199673Z OK (skipped=1)
2022-11-23T02:08:16.2199691Z 
2022-11-23T02:08:16.2199811Z Generating XML reports...
2022-11-23T02:08:16.2200245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015045.xml
2022-11-23T02:08:16.2200610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2200779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2201154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2201336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2201355Z 
2022-11-23T02:08:16.2201446Z Running tests...
2022-11-23T02:08:16.2201707Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2202060Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2202334Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2203078Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.679s)
2022-11-23T02:08:16.2203101Z 
2022-11-23T02:08:16.2203353Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2203509Z Ran 1 test in 1.679s
2022-11-23T02:08:16.2203529Z 
2022-11-23T02:08:16.2203630Z OK (skipped=1)
2022-11-23T02:08:16.2203649Z 
2022-11-23T02:08:16.2203772Z Generating XML reports...
2022-11-23T02:08:16.2204207Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015047.xml
2022-11-23T02:08:16.2204564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2204739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2205116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2205299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2205319Z 
2022-11-23T02:08:16.2205425Z Running tests...
2022-11-23T02:08:16.2205679Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2205987Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2206266Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2206470Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26283
2022-11-23T02:08:16.2206692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26284
2022-11-23T02:08:16.2207050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2207223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2207591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2207833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2208188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2208357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2208720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2208897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2209145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2209388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2209776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2210167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2210395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2210615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2210948Z STAGE:2022-11-23 01:50:57 26284:26284 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2211307Z STAGE:2022-11-23 01:50:57 26283:26283 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2211981Z STAGE:2022-11-23 01:50:57 26283:26283 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2212330Z STAGE:2022-11-23 01:50:57 26284:26284 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2212902Z STAGE:2022-11-23 01:50:57 26283:26283 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:50:57 26284:26284 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2212924Z 
2022-11-23T02:08:16.2213017Z ok (6.481s)
2022-11-23T02:08:16.2213036Z 
2022-11-23T02:08:16.2213297Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2213401Z Ran 1 test in 6.482s
2022-11-23T02:08:16.2213421Z 
2022-11-23T02:08:16.2213510Z OK
2022-11-23T02:08:16.2213529Z 
2022-11-23T02:08:16.2213643Z Generating XML reports...
2022-11-23T02:08:16.2214098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015052.xml
2022-11-23T02:08:16.2214453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2214621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2215000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2215181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2215200Z 
2022-11-23T02:08:16.2215307Z Running tests...
2022-11-23T02:08:16.2215570Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2215873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2216173Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:08:16.2216193Z 
2022-11-23T02:08:16.2216444Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2216539Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2216558Z 
2022-11-23T02:08:16.2216665Z OK (skipped=1)
2022-11-23T02:08:16.2216683Z 
2022-11-23T02:08:16.2216796Z Generating XML reports...
2022-11-23T02:08:16.2217341Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015101.xml
2022-11-23T02:08:16.2217701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2217878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2218256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2218442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2218462Z 
2022-11-23T02:08:16.2218569Z Running tests...
2022-11-23T02:08:16.2218815Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2219119Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2219408Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2220145Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.647s)
2022-11-23T02:08:16.2220165Z 
2022-11-23T02:08:16.2220428Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2220536Z Ran 1 test in 1.648s
2022-11-23T02:08:16.2220555Z 
2022-11-23T02:08:16.2220716Z OK (skipped=1)
2022-11-23T02:08:16.2220738Z 
2022-11-23T02:08:16.2220865Z Generating XML reports...
2022-11-23T02:08:16.2221310Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015103.xml
2022-11-23T02:08:16.2221660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2221837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2222208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2222400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2222419Z 
2022-11-23T02:08:16.2222525Z Running tests...
2022-11-23T02:08:16.2222778Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2223092Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2223374Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:08:16.2223394Z 
2022-11-23T02:08:16.2223652Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2223745Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2223774Z 
2022-11-23T02:08:16.2223865Z OK (skipped=1)
2022-11-23T02:08:16.2223883Z 
2022-11-23T02:08:16.2224001Z Generating XML reports...
2022-11-23T02:08:16.2224438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015107.xml
2022-11-23T02:08:16.2224804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2224970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2225349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2225537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2225556Z 
2022-11-23T02:08:16.2225665Z Running tests...
2022-11-23T02:08:16.2225913Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2226227Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2226569Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2226789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26504
2022-11-23T02:08:16.2227009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26505
2022-11-23T02:08:16.2227378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2227554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2227935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2228109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2228470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2228646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2229020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2229208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2229452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2229762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2230175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2230575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2230790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2231025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2231187Z skip: Skipped due to small world size. (4.121s)
2022-11-23T02:08:16.2231206Z 
2022-11-23T02:08:16.2231471Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2231582Z Ran 1 test in 4.121s
2022-11-23T02:08:16.2231601Z 
2022-11-23T02:08:16.2231707Z OK (skipped=1)
2022-11-23T02:08:16.2231726Z 
2022-11-23T02:08:16.2231849Z Generating XML reports...
2022-11-23T02:08:16.2232307Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015110.xml
2022-11-23T02:08:16.2232680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2232836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2233220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2233408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2233427Z 
2022-11-23T02:08:16.2233536Z Running tests...
2022-11-23T02:08:16.2233801Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2234114Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2234403Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:08:16.2234423Z 
2022-11-23T02:08:16.2234683Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2234793Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2234812Z 
2022-11-23T02:08:16.2234904Z OK (skipped=1)
2022-11-23T02:08:16.2234922Z 
2022-11-23T02:08:16.2235045Z Generating XML reports...
2022-11-23T02:08:16.2235556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015116.xml
2022-11-23T02:08:16.2235925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2236099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2236474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2236666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2236686Z 
2022-11-23T02:08:16.2236796Z Running tests...
2022-11-23T02:08:16.2237041Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2237353Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2237651Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:08:16.2237674Z 
2022-11-23T02:08:16.2237932Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2238043Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2238061Z 
2022-11-23T02:08:16.2238168Z OK (skipped=1)
2022-11-23T02:08:16.2238187Z 
2022-11-23T02:08:16.2238310Z Generating XML reports...
2022-11-23T02:08:16.2238756Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015119.xml
2022-11-23T02:08:16.2239173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2239338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2239723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2239914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2239937Z 
2022-11-23T02:08:16.2240047Z Running tests...
2022-11-23T02:08:16.2240309Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2240618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2240899Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2241120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26675
2022-11-23T02:08:16.2241344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26676
2022-11-23T02:08:16.2241696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2241871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2242250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2242442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2242805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2242980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2243351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2243541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2243770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2244013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2244416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2244871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2245103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2245332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2245436Z ok (5.650s)
2022-11-23T02:08:16.2245455Z 
2022-11-23T02:08:16.2245720Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2245836Z Ran 1 test in 5.650s
2022-11-23T02:08:16.2245855Z 
2022-11-23T02:08:16.2245932Z OK
2022-11-23T02:08:16.2245951Z 
2022-11-23T02:08:16.2246074Z Generating XML reports...
2022-11-23T02:08:16.2246520Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015121.xml
2022-11-23T02:08:16.2246888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2247066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2247445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2247636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2247656Z 
2022-11-23T02:08:16.2247764Z Running tests...
2022-11-23T02:08:16.2248007Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2248367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2248669Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2248888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26792
2022-11-23T02:08:16.2249109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26793
2022-11-23T02:08:16.2249479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2249654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2250033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2250222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2250570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2250742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2251111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2251297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2251795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2252051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2252456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2252851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2253087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2253299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2253403Z ok (5.516s)
2022-11-23T02:08:16.2253423Z 
2022-11-23T02:08:16.2253690Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2253888Z Ran 1 test in 5.516s
2022-11-23T02:08:16.2253907Z 
2022-11-23T02:08:16.2254000Z OK
2022-11-23T02:08:16.2254019Z 
2022-11-23T02:08:16.2254143Z Generating XML reports...
2022-11-23T02:08:16.2254599Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015129.xml
2022-11-23T02:08:16.2254971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2255130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2255515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2255703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2255722Z 
2022-11-23T02:08:16.2255831Z Running tests...
2022-11-23T02:08:16.2256091Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2256409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2256710Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:08:16.2256730Z 
2022-11-23T02:08:16.2256988Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2257099Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2257118Z 
2022-11-23T02:08:16.2257208Z OK (skipped=1)
2022-11-23T02:08:16.2257227Z 
2022-11-23T02:08:16.2257351Z Generating XML reports...
2022-11-23T02:08:16.2257866Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015137.xml
2022-11-23T02:08:16.2258251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2258431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2259035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2259505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2259738Z 
2022-11-23T02:08:16.2259852Z Running tests...
2022-11-23T02:08:16.2260258Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2260798Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2261332Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2261851Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26942
2022-11-23T02:08:16.2262306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26943
2022-11-23T02:08:16.2262913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2263351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2263931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2264399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2264960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2265415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2265994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2266464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2266898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2267406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2268132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2268817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2269324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2269861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.2270348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2270840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.2271477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2272166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2272564Z ok (5.657s)
2022-11-23T02:08:16.2272719Z 
2022-11-23T02:08:16.2272993Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2273315Z Ran 1 test in 5.657s
2022-11-23T02:08:16.2273481Z 
2022-11-23T02:08:16.2273582Z OK
2022-11-23T02:08:16.2273720Z 
2022-11-23T02:08:16.2273848Z Generating XML reports...
2022-11-23T02:08:16.2276646Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015140.xml
2022-11-23T02:08:16.2277520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2277994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2278584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2279066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2279283Z 
2022-11-23T02:08:16.2279392Z Running tests...
2022-11-23T02:08:16.2279804Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2280345Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2280880Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:08:16.2281218Z 
2022-11-23T02:08:16.2281490Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2281823Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2281986Z 
2022-11-23T02:08:16.2282098Z OK (skipped=1)
2022-11-23T02:08:16.2282234Z 
2022-11-23T02:08:16.2282362Z Generating XML reports...
2022-11-23T02:08:16.2282977Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015148.xml
2022-11-23T02:08:16.2283724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2284178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2284745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2285231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2285472Z 
2022-11-23T02:08:16.2285587Z Running tests...
2022-11-23T02:08:16.2285990Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2286526Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2287073Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2287683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27094
2022-11-23T02:08:16.2288119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27095
2022-11-23T02:08:16.2288730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2289184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2289763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2290223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2290798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2291240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2292200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2292682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2293140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2293644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2294314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2295164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2295732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2296204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2296594Z skip: Skipped due to small world size. (4.230s)
2022-11-23T02:08:16.2296801Z 
2022-11-23T02:08:16.2297081Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2297415Z Ran 1 test in 4.230s
2022-11-23T02:08:16.2297579Z 
2022-11-23T02:08:16.2297687Z OK (skipped=1)
2022-11-23T02:08:16.2297841Z 
2022-11-23T02:08:16.2297950Z Generating XML reports...
2022-11-23T02:08:16.2298561Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015150.xml
2022-11-23T02:08:16.2299284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2299739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2300303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2300780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2301019Z 
2022-11-23T02:08:16.2301134Z Running tests...
2022-11-23T02:08:16.2301523Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2302062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2302611Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2303194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27199
2022-11-23T02:08:16.2303638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27200
2022-11-23T02:08:16.2304274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2304732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2305315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2305893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2306496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2306956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2307535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2307991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2308456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2308968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2309635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2310318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2310843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2311316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2311787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.2312342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.2313012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2313697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2314072Z ok (5.672s)
2022-11-23T02:08:16.2314227Z 
2022-11-23T02:08:16.2314496Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2314825Z Ran 1 test in 5.672s
2022-11-23T02:08:16.2314987Z 
2022-11-23T02:08:16.2315065Z OK
2022-11-23T02:08:16.2315202Z 
2022-11-23T02:08:16.2315328Z Generating XML reports...
2022-11-23T02:08:16.2315940Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015157.xml
2022-11-23T02:08:16.2316661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2317102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2317681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2318158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2318389Z 
2022-11-23T02:08:16.2318501Z Running tests...
2022-11-23T02:08:16.2318896Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2319426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2319938Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2320414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27323
2022-11-23T02:08:16.2320870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27324
2022-11-23T02:08:16.2321480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2321933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2322491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2322963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2323617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2324046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2324620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2325081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2325542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2326029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2326683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2327372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2327904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2328363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2328752Z skip: Need at least 3 CUDA devices (4.211s)
2022-11-23T02:08:16.2328948Z 
2022-11-23T02:08:16.2329217Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2329529Z Ran 1 test in 4.211s
2022-11-23T02:08:16.2329693Z 
2022-11-23T02:08:16.2329803Z OK (skipped=1)
2022-11-23T02:08:16.2330013Z 
2022-11-23T02:08:16.2330147Z Generating XML reports...
2022-11-23T02:08:16.2330763Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015205.xml
2022-11-23T02:08:16.2331672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2332149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2332735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2333190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2333424Z 
2022-11-23T02:08:16.2333535Z Running tests...
2022-11-23T02:08:16.2333938Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2334473Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2334971Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-11-23T02:08:16.2335257Z 
2022-11-23T02:08:16.2335523Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2335854Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2336016Z 
2022-11-23T02:08:16.2336106Z OK (skipped=1)
2022-11-23T02:08:16.2336265Z 
2022-11-23T02:08:16.2336390Z Generating XML reports...
2022-11-23T02:08:16.2336996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015212.xml
2022-11-23T02:08:16.2337716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2338151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2338724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2339198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2339429Z 
2022-11-23T02:08:16.2339539Z Running tests...
2022-11-23T02:08:16.2339927Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2340461Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2341071Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T02:08:16.2341359Z 
2022-11-23T02:08:16.2341613Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2341939Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2342102Z 
2022-11-23T02:08:16.2342214Z OK (skipped=1)
2022-11-23T02:08:16.2342370Z 
2022-11-23T02:08:16.2342496Z Generating XML reports...
2022-11-23T02:08:16.2343089Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015214.xml
2022-11-23T02:08:16.2343817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2344269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2344828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2345308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2345539Z 
2022-11-23T02:08:16.2345651Z Running tests...
2022-11-23T02:08:16.2346057Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2346582Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2347084Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2347570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27494
2022-11-23T02:08:16.2348080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27495
2022-11-23T02:08:16.2348712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2349163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2349739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2350200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2350778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2351222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2351771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2352244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2352699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2353204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2353853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2354547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2355071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2355546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2355881Z ok (6.327s)
2022-11-23T02:08:16.2356030Z 
2022-11-23T02:08:16.2356302Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2356638Z Ran 1 test in 6.327s
2022-11-23T02:08:16.2356804Z 
2022-11-23T02:08:16.2356882Z OK
2022-11-23T02:08:16.2357019Z 
2022-11-23T02:08:16.2357145Z Generating XML reports...
2022-11-23T02:08:16.2357753Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015217.xml
2022-11-23T02:08:16.2358471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2358982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2359559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2360035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2360266Z 
2022-11-23T02:08:16.2360358Z Running tests...
2022-11-23T02:08:16.2360762Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2361296Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2361812Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T02:08:16.2362108Z 
2022-11-23T02:08:16.2362356Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2362687Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2362850Z 
2022-11-23T02:08:16.2362960Z OK (skipped=1)
2022-11-23T02:08:16.2363114Z 
2022-11-23T02:08:16.2363219Z Generating XML reports...
2022-11-23T02:08:16.2363831Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015226.xml
2022-11-23T02:08:16.2364554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2365007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2365625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2366118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2366351Z 
2022-11-23T02:08:16.2366465Z Running tests...
2022-11-23T02:08:16.2366871Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2367387Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2367908Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2368410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27642
2022-11-23T02:08:16.2368849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27643
2022-11-23T02:08:16.2369457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2369912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2370480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2370915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2371736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2372232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2372812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2373284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2373744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2374252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2374898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2375590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2376219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2376698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2377088Z skip: Skipped due to small world size. (4.228s)
2022-11-23T02:08:16.2377290Z 
2022-11-23T02:08:16.2377564Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2377900Z Ran 1 test in 4.229s
2022-11-23T02:08:16.2378064Z 
2022-11-23T02:08:16.2378156Z OK (skipped=1)
2022-11-23T02:08:16.2378313Z 
2022-11-23T02:08:16.2378440Z Generating XML reports...
2022-11-23T02:08:16.2379052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015228.xml
2022-11-23T02:08:16.2379772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2380210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2380792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2381262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2381494Z 
2022-11-23T02:08:16.2381588Z Running tests...
2022-11-23T02:08:16.2381991Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2382522Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2383106Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-11-23T02:08:16.2383393Z 
2022-11-23T02:08:16.2383669Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2383994Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2384157Z 
2022-11-23T02:08:16.2384267Z OK (skipped=1)
2022-11-23T02:08:16.2384420Z 
2022-11-23T02:08:16.2384527Z Generating XML reports...
2022-11-23T02:08:16.2385142Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015235.xml
2022-11-23T02:08:16.2385857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2386313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2386876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2387347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2387578Z 
2022-11-23T02:08:16.2387688Z Running tests...
2022-11-23T02:08:16.2388075Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2388611Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2389124Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2389625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27780
2022-11-23T02:08:16.2390062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27781
2022-11-23T02:08:16.2390666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2391117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2391696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2392155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2392737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2393182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2393732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2394285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2394740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2395248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2395897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2396587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2397114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2397588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2397981Z skip: Skipped due to small world size. (4.230s)
2022-11-23T02:08:16.2398183Z 
2022-11-23T02:08:16.2398450Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2398780Z Ran 1 test in 4.231s
2022-11-23T02:08:16.2398940Z 
2022-11-23T02:08:16.2399031Z OK (skipped=1)
2022-11-23T02:08:16.2399182Z 
2022-11-23T02:08:16.2399309Z Generating XML reports...
2022-11-23T02:08:16.2399920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015237.xml
2022-11-23T02:08:16.2400742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2401191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2401767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2402237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2402473Z 
2022-11-23T02:08:16.2402567Z Running tests...
2022-11-23T02:08:16.2402970Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2403559Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2404090Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T02:08:16.2404380Z 
2022-11-23T02:08:16.2404646Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2404979Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2405142Z 
2022-11-23T02:08:16.2405251Z OK (skipped=1)
2022-11-23T02:08:16.2405409Z 
2022-11-23T02:08:16.2405516Z Generating XML reports...
2022-11-23T02:08:16.2406122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015244.xml
2022-11-23T02:08:16.2406835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2407287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2407845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2408315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2408545Z 
2022-11-23T02:08:16.2408657Z Running tests...
2022-11-23T02:08:16.2409045Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2409580Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2410100Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T02:08:16.2410401Z 
2022-11-23T02:08:16.2410663Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2411041Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2411204Z 
2022-11-23T02:08:16.2411313Z OK (skipped=1)
2022-11-23T02:08:16.2411714Z 
2022-11-23T02:08:16.2411855Z Generating XML reports...
2022-11-23T02:08:16.2412475Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015246.xml
2022-11-23T02:08:16.2413173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2413628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2414203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2414656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2414887Z 
2022-11-23T02:08:16.2414999Z Running tests...
2022-11-23T02:08:16.2415401Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2415936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2416438Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-11-23T02:08:16.2416741Z 
2022-11-23T02:08:16.2417006Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2417335Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2417497Z 
2022-11-23T02:08:16.2417589Z OK (skipped=1)
2022-11-23T02:08:16.2417747Z 
2022-11-23T02:08:16.2417872Z Generating XML reports...
2022-11-23T02:08:16.2418569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015249.xml
2022-11-23T02:08:16.2419306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2419744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2420325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2420797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2421030Z 
2022-11-23T02:08:16.2421140Z Running tests...
2022-11-23T02:08:16.2421528Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2422057Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2422569Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-11-23T02:08:16.2422860Z 
2022-11-23T02:08:16.2423106Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2423432Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2423594Z 
2022-11-23T02:08:16.2423703Z OK (skipped=1)
2022-11-23T02:08:16.2423857Z 
2022-11-23T02:08:16.2423983Z Generating XML reports...
2022-11-23T02:08:16.2424576Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015251.xml
2022-11-23T02:08:16.2425299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2425752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2426313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2426784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2427018Z 
2022-11-23T02:08:16.2427128Z Running tests...
2022-11-23T02:08:16.2427530Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2428042Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2428553Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-11-23T02:08:16.2428938Z 
2022-11-23T02:08:16.2429204Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2429511Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2429673Z 
2022-11-23T02:08:16.2429782Z OK (skipped=1)
2022-11-23T02:08:16.2429937Z 
2022-11-23T02:08:16.2430062Z Generating XML reports...
2022-11-23T02:08:16.2430666Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015254.xml
2022-11-23T02:08:16.2431365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2431814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2432389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2432842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2433075Z 
2022-11-23T02:08:16.2433185Z Running tests...
2022-11-23T02:08:16.2433584Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2434115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2434630Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2435141Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28050
2022-11-23T02:08:16.2435654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28051
2022-11-23T02:08:16.2436279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2436714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2437289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2437767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2438326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2438770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2439336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2439801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2440245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2440746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2441403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2442082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2442607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2443085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2443570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.2444057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.2444716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2445400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2445930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:08:16.2446480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:08:16.2447135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.2447818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.2448210Z ok (4.344s)
2022-11-23T02:08:16.2448343Z 
2022-11-23T02:08:16.2448614Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2448944Z Ran 1 test in 4.344s
2022-11-23T02:08:16.2449106Z 
2022-11-23T02:08:16.2449203Z OK
2022-11-23T02:08:16.2449338Z 
2022-11-23T02:08:16.2449447Z Generating XML reports...
2022-11-23T02:08:16.2450063Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015256.xml
2022-11-23T02:08:16.2450782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2451237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2452060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2452533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2452766Z 
2022-11-23T02:08:16.2452878Z Running tests...
2022-11-23T02:08:16.2453348Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2453907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2454425Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2454926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28163
2022-11-23T02:08:16.2455368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28164
2022-11-23T02:08:16.2455977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2456428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2456991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2457467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2458052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2458498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2459053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2459531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2459986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2460490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2461133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2461824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2462353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2462813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2463160Z ok (5.253s)
2022-11-23T02:08:16.2463311Z 
2022-11-23T02:08:16.2463580Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2464000Z Ran 1 test in 5.253s
2022-11-23T02:08:16.2464144Z 
2022-11-23T02:08:16.2464240Z OK
2022-11-23T02:08:16.2464376Z 
2022-11-23T02:08:16.2464505Z Generating XML reports...
2022-11-23T02:08:16.2465115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015303.xml
2022-11-23T02:08:16.2465816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2466273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2466859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2467331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2467569Z 
2022-11-23T02:08:16.2467662Z Running tests...
2022-11-23T02:08:16.2468068Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2468609Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2469130Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2469638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28279
2022-11-23T02:08:16.2470095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28280
2022-11-23T02:08:16.2470767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2471214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2471802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2472276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2472861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2473293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2473867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2474343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2474783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2475297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2475960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2476655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2477157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2477638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2478038Z skip: Skipped due to small world size. (4.208s)
2022-11-23T02:08:16.2478238Z 
2022-11-23T02:08:16.2478513Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2478830Z Ran 1 test in 4.208s
2022-11-23T02:08:16.2478991Z 
2022-11-23T02:08:16.2479102Z OK (skipped=1)
2022-11-23T02:08:16.2479261Z 
2022-11-23T02:08:16.2479389Z Generating XML reports...
2022-11-23T02:08:16.2479983Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015311.xml
2022-11-23T02:08:16.2480718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2481168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2481822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2482282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2482512Z 
2022-11-23T02:08:16.2482621Z Running tests...
2022-11-23T02:08:16.2483029Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2483562Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2484067Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2484565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28384
2022-11-23T02:08:16.2485021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28385
2022-11-23T02:08:16.2485615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2486070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2486646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2487119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2487679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2488128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2488751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2489215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2489671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2490178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2490847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2491759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2492302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2492788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2493136Z ok (5.056s)
2022-11-23T02:08:16.2493267Z 
2022-11-23T02:08:16.2493544Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2493881Z Ran 1 test in 5.056s
2022-11-23T02:08:16.2494044Z 
2022-11-23T02:08:16.2494142Z OK
2022-11-23T02:08:16.2494285Z 
2022-11-23T02:08:16.2494392Z Generating XML reports...
2022-11-23T02:08:16.2495010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015317.xml
2022-11-23T02:08:16.2495735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2496192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2496760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2497236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2497467Z 
2022-11-23T02:08:16.2497583Z Running tests...
2022-11-23T02:08:16.2497986Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2498505Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2499032Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2499652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28490
2022-11-23T02:08:16.2500089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28491
2022-11-23T02:08:16.2500707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2501164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2501748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2502209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2502792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2503301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2503869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2504347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2504807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2505318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2506025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2506747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2507277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2507759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2508091Z ok (4.348s)
2022-11-23T02:08:16.2508242Z 
2022-11-23T02:08:16.2508506Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2508840Z Ran 1 test in 4.348s
2022-11-23T02:08:16.2508859Z 
2022-11-23T02:08:16.2508950Z OK
2022-11-23T02:08:16.2508969Z 
2022-11-23T02:08:16.2509095Z Generating XML reports...
2022-11-23T02:08:16.2509529Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015325.xml
2022-11-23T02:08:16.2509913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2510089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2510473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2510661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2510683Z 
2022-11-23T02:08:16.2510792Z Running tests...
2022-11-23T02:08:16.2511057Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2511371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2511657Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2511860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28595
2022-11-23T02:08:16.2512081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28596
2022-11-23T02:08:16.2512454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2512633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2513012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2513287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2513654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2513832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2514187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2514377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2514632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2514877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2515276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2515680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2515914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2516142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2516243Z ok (5.206s)
2022-11-23T02:08:16.2516263Z 
2022-11-23T02:08:16.2516511Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2516627Z Ran 1 test in 5.206s
2022-11-23T02:08:16.2516697Z 
2022-11-23T02:08:16.2516800Z OK
2022-11-23T02:08:16.2516819Z 
2022-11-23T02:08:16.2516947Z Generating XML reports...
2022-11-23T02:08:16.2517406Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015332.xml
2022-11-23T02:08:16.2517783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2517963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2518341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2518535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2518554Z 
2022-11-23T02:08:16.2518647Z Running tests...
2022-11-23T02:08:16.2518909Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2519227Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2519501Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2519724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28711
2022-11-23T02:08:16.2519942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28712
2022-11-23T02:08:16.2520319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2520494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2520878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2521052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2521416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2521589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2521966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2522150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2522461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2522711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2523113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2523509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2523725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2523957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2524058Z ok (5.300s)
2022-11-23T02:08:16.2524078Z 
2022-11-23T02:08:16.2524345Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2524456Z Ran 1 test in 5.300s
2022-11-23T02:08:16.2524478Z 
2022-11-23T02:08:16.2524570Z OK
2022-11-23T02:08:16.2524589Z 
2022-11-23T02:08:16.2524712Z Generating XML reports...
2022-11-23T02:08:16.2525163Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015340.xml
2022-11-23T02:08:16.2525519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2525700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2526131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2526329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2526348Z 
2022-11-23T02:08:16.2526457Z Running tests...
2022-11-23T02:08:16.2526723Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2527037Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2527317Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2527539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28825
2022-11-23T02:08:16.2527739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28826
2022-11-23T02:08:16.2528115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2528289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2528673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2528861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2529229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2529403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2529781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2529970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2530199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2530446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2530854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2531250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2531684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2532023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2532127Z ok (4.375s)
2022-11-23T02:08:16.2532147Z 
2022-11-23T02:08:16.2532419Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2532515Z Ran 1 test in 4.375s
2022-11-23T02:08:16.2532553Z 
2022-11-23T02:08:16.2532629Z OK
2022-11-23T02:08:16.2532648Z 
2022-11-23T02:08:16.2532775Z Generating XML reports...
2022-11-23T02:08:16.2533234Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015347.xml
2022-11-23T02:08:16.2533612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2533791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2534172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2534365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2534384Z 
2022-11-23T02:08:16.2534493Z Running tests...
2022-11-23T02:08:16.2534737Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2535051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2535306Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2535326Z 
2022-11-23T02:08:16.2535660Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2535784Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2535803Z 
2022-11-23T02:08:16.2535913Z OK (skipped=1)
2022-11-23T02:08:16.2535932Z 
2022-11-23T02:08:16.2536060Z Generating XML reports...
2022-11-23T02:08:16.2536516Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015354.xml
2022-11-23T02:08:16.2536892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2537051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2537431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2537622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2537641Z 
2022-11-23T02:08:16.2537750Z Running tests...
2022-11-23T02:08:16.2538016Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2538334Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2538590Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2539356Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2022-11-23T02:08:16.2539380Z 
2022-11-23T02:08:16.2539646Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2539761Z Ran 1 test in 1.641s
2022-11-23T02:08:16.2539780Z 
2022-11-23T02:08:16.2539870Z OK (skipped=1)
2022-11-23T02:08:16.2539888Z 
2022-11-23T02:08:16.2540014Z Generating XML reports...
2022-11-23T02:08:16.2540467Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015357.xml
2022-11-23T02:08:16.2540839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2541019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2541395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2541651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2541670Z 
2022-11-23T02:08:16.2541780Z Running tests...
2022-11-23T02:08:16.2542025Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2542344Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2542620Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2542639Z 
2022-11-23T02:08:16.2542900Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2543016Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2543035Z 
2022-11-23T02:08:16.2543142Z OK (skipped=1)
2022-11-23T02:08:16.2543161Z 
2022-11-23T02:08:16.2543287Z Generating XML reports...
2022-11-23T02:08:16.2543734Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015401.xml
2022-11-23T02:08:16.2544108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2544266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2544646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2544838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2544857Z 
2022-11-23T02:08:16.2545014Z Running tests...
2022-11-23T02:08:16.2545285Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2545602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2545874Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.2545897Z 
2022-11-23T02:08:16.2546162Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2546273Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2546293Z 
2022-11-23T02:08:16.2546384Z OK (skipped=1)
2022-11-23T02:08:16.2546421Z 
2022-11-23T02:08:16.2546528Z Generating XML reports...
2022-11-23T02:08:16.2546979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015403.xml
2022-11-23T02:08:16.2547353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2547528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2547909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2548098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2548120Z 
2022-11-23T02:08:16.2548233Z Running tests...
2022-11-23T02:08:16.2548499Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2548796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2549066Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-11-23T02:08:16.2549086Z 
2022-11-23T02:08:16.2549348Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2549460Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2549479Z 
2022-11-23T02:08:16.2549593Z OK (skipped=1)
2022-11-23T02:08:16.2549612Z 
2022-11-23T02:08:16.2549737Z Generating XML reports...
2022-11-23T02:08:16.2550189Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015406.xml
2022-11-23T02:08:16.2550562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2550803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2551167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2551365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2551385Z 
2022-11-23T02:08:16.2551494Z Running tests...
2022-11-23T02:08:16.2551758Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2552074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2552343Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2553097Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.656s)
2022-11-23T02:08:16.2553120Z 
2022-11-23T02:08:16.2553383Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2553500Z Ran 1 test in 1.656s
2022-11-23T02:08:16.2553519Z 
2022-11-23T02:08:16.2553609Z OK (skipped=1)
2022-11-23T02:08:16.2553646Z 
2022-11-23T02:08:16.2553753Z Generating XML reports...
2022-11-23T02:08:16.2554203Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015408.xml
2022-11-23T02:08:16.2554626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2554814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2555203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2555402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2555421Z 
2022-11-23T02:08:16.2555533Z Running tests...
2022-11-23T02:08:16.2555797Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2556090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2556408Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2557161Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.600s)
2022-11-23T02:08:16.2557182Z 
2022-11-23T02:08:16.2557445Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2557562Z Ran 1 test in 1.601s
2022-11-23T02:08:16.2557581Z 
2022-11-23T02:08:16.2557691Z OK (skipped=1)
2022-11-23T02:08:16.2557709Z 
2022-11-23T02:08:16.2557834Z Generating XML reports...
2022-11-23T02:08:16.2558288Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015412.xml
2022-11-23T02:08:16.2558660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2558840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2559208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2559401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2559420Z 
2022-11-23T02:08:16.2559529Z Running tests...
2022-11-23T02:08:16.2559790Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2560168Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2560486Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2561230Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-11-23T02:08:16.2561253Z 
2022-11-23T02:08:16.2561515Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2561629Z Ran 1 test in 1.645s
2022-11-23T02:08:16.2561648Z 
2022-11-23T02:08:16.2561755Z OK (skipped=1)
2022-11-23T02:08:16.2561775Z 
2022-11-23T02:08:16.2561882Z Generating XML reports...
2022-11-23T02:08:16.2562332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015416.xml
2022-11-23T02:08:16.2562709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2562888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2563269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2563461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2563480Z 
2022-11-23T02:08:16.2563641Z Running tests...
2022-11-23T02:08:16.2563917Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2564213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2564486Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2564714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29198
2022-11-23T02:08:16.2564935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29199
2022-11-23T02:08:16.2565308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2565486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2565867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2566062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2566431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2566588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2566963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2567158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2567403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2567653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2568060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2568463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2568695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2568927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2569169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagmm2sq4
2022-11-23T02:08:16.2569513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagmm2sq4/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2569771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpnjxqnv
2022-11-23T02:08:16.2570046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpnjxqnv/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2570286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2570528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2570633Z ok (5.647s)
2022-11-23T02:08:16.2570652Z 
2022-11-23T02:08:16.2570924Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2571019Z Ran 1 test in 5.647s
2022-11-23T02:08:16.2571058Z 
2022-11-23T02:08:16.2571134Z OK
2022-11-23T02:08:16.2571153Z 
2022-11-23T02:08:16.2571278Z Generating XML reports...
2022-11-23T02:08:16.2572048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015421.xml
2022-11-23T02:08:16.2572435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2572614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2572993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2573270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2573293Z 
2022-11-23T02:08:16.2573414Z Running tests...
2022-11-23T02:08:16.2573663Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2573981Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2574262Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2574488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29316
2022-11-23T02:08:16.2574712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29317
2022-11-23T02:08:16.2575085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2575262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2575703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2575901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2576252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2576430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2576807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2576995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2577244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2577488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2577898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2578300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2578533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2578745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2579111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0se4dcb
2022-11-23T02:08:16.2579382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0se4dcb/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2579635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcq6uifea
2022-11-23T02:08:16.2579907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcq6uifea/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2580152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2580387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2580626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2580843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2580951Z ok (5.620s)
2022-11-23T02:08:16.2580970Z 
2022-11-23T02:08:16.2581242Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2581362Z Ran 1 test in 5.621s
2022-11-23T02:08:16.2581382Z 
2022-11-23T02:08:16.2581477Z OK
2022-11-23T02:08:16.2581497Z 
2022-11-23T02:08:16.2581622Z Generating XML reports...
2022-11-23T02:08:16.2582080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015429.xml
2022-11-23T02:08:16.2582510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2582700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2583070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2583267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2583291Z 
2022-11-23T02:08:16.2583400Z Running tests...
2022-11-23T02:08:16.2583663Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2583977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2584255Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2585008Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.626s)
2022-11-23T02:08:16.2585029Z 
2022-11-23T02:08:16.2585289Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2585408Z Ran 1 test in 1.627s
2022-11-23T02:08:16.2585427Z 
2022-11-23T02:08:16.2585517Z OK (skipped=1)
2022-11-23T02:08:16.2585559Z 
2022-11-23T02:08:16.2585666Z Generating XML reports...
2022-11-23T02:08:16.2586120Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015437.xml
2022-11-23T02:08:16.2586491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2586666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2587045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2587243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2587262Z 
2022-11-23T02:08:16.2587371Z Running tests...
2022-11-23T02:08:16.2587637Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2587933Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2588293Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2589043Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.647s)
2022-11-23T02:08:16.2589064Z 
2022-11-23T02:08:16.2589328Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2589444Z Ran 1 test in 1.647s
2022-11-23T02:08:16.2589463Z 
2022-11-23T02:08:16.2589574Z OK (skipped=1)
2022-11-23T02:08:16.2589593Z 
2022-11-23T02:08:16.2589719Z Generating XML reports...
2022-11-23T02:08:16.2590172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015441.xml
2022-11-23T02:08:16.2590546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2590729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2591090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2591281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2591301Z 
2022-11-23T02:08:16.2591409Z Running tests...
2022-11-23T02:08:16.2591672Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2592035Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2592335Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2592560Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29502
2022-11-23T02:08:16.2592780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29503
2022-11-23T02:08:16.2593159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2593317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2593698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2593891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2594264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2594438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2594807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2595001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2595252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2595480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2595885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2596285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2596521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2596752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2597009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50vaszfu
2022-11-23T02:08:16.2597283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50vaszfu/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2597639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpef7h6kuc
2022-11-23T02:08:16.2597914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpef7h6kuc/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2598110Z <class 'torch.nn.parameter.Parameter'>
2022-11-23T02:08:16.2598215Z ok (5.764s)
2022-11-23T02:08:16.2598234Z 
2022-11-23T02:08:16.2598500Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2598616Z Ran 1 test in 5.765s
2022-11-23T02:08:16.2598636Z 
2022-11-23T02:08:16.2598733Z OK
2022-11-23T02:08:16.2598751Z 
2022-11-23T02:08:16.2598875Z Generating XML reports...
2022-11-23T02:08:16.2599331Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015445.xml
2022-11-23T02:08:16.2599700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2599886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2600252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2600445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2600464Z 
2022-11-23T02:08:16.2600572Z Running tests...
2022-11-23T02:08:16.2600837Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2601202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2601519Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2601744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29617
2022-11-23T02:08:16.2601965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29618
2022-11-23T02:08:16.2602324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2602501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2602881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2603072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2603499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2603675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2604054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2604244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2604498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2604727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2605132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2605528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2605767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2605996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2606262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu289tofn
2022-11-23T02:08:16.2606531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu289tofn/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2606856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lv07sw0
2022-11-23T02:08:16.2607120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lv07sw0/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2607205Z ok (5.768s)
2022-11-23T02:08:16.2607225Z 
2022-11-23T02:08:16.2607497Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2607613Z Ran 1 test in 5.768s
2022-11-23T02:08:16.2607633Z 
2022-11-23T02:08:16.2607729Z OK
2022-11-23T02:08:16.2607748Z 
2022-11-23T02:08:16.2607871Z Generating XML reports...
2022-11-23T02:08:16.2608330Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015454.xml
2022-11-23T02:08:16.2608703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2608883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2609252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2609446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2609465Z 
2022-11-23T02:08:16.2609574Z Running tests...
2022-11-23T02:08:16.2609842Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2610156Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2610468Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2610698Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29732
2022-11-23T02:08:16.2610924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29733
2022-11-23T02:08:16.2611299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2611704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2612103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2612295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2612662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2612840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2613219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2613408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2613656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2613902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2614295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2614690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2614921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2615150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2615411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56hhhnzi
2022-11-23T02:08:16.2615681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56hhhnzi/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2615939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1xsrarg
2022-11-23T02:08:16.2616303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1xsrarg/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2616523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2616759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2616860Z ok (6.028s)
2022-11-23T02:08:16.2616880Z 
2022-11-23T02:08:16.2617153Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2617265Z Ran 1 test in 6.028s
2022-11-23T02:08:16.2617285Z 
2022-11-23T02:08:16.2617379Z OK
2022-11-23T02:08:16.2617402Z 
2022-11-23T02:08:16.2617527Z Generating XML reports...
2022-11-23T02:08:16.2617981Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015502.xml
2022-11-23T02:08:16.2618354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2618516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2618897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2619092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2619111Z 
2022-11-23T02:08:16.2619223Z Running tests...
2022-11-23T02:08:16.2619484Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2619799Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2620163Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2620397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29851
2022-11-23T02:08:16.2620598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29852
2022-11-23T02:08:16.2620979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2621155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2621534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2621727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2622091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2622267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2622640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2622833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2623062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2623312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2623715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2624110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2624342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2624572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2624831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmgb8llh
2022-11-23T02:08:16.2625107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmgb8llh/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2625364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoapy015f
2022-11-23T02:08:16.2625674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoapy015f/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2626454Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:08:16.2626562Z ok (5.663s)
2022-11-23T02:08:16.2626581Z 
2022-11-23T02:08:16.2626855Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2626968Z Ran 1 test in 5.664s
2022-11-23T02:08:16.2626987Z 
2022-11-23T02:08:16.2627085Z OK
2022-11-23T02:08:16.2627104Z 
2022-11-23T02:08:16.2627231Z Generating XML reports...
2022-11-23T02:08:16.2627684Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015510.xml
2022-11-23T02:08:16.2628053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2628229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2628641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2628844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2628864Z 
2022-11-23T02:08:16.2628971Z Running tests...
2022-11-23T02:08:16.2629234Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2629550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2629838Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2630590Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2022-11-23T02:08:16.2630611Z 
2022-11-23T02:08:16.2630872Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2630991Z Ran 1 test in 1.635s
2022-11-23T02:08:16.2631010Z 
2022-11-23T02:08:16.2631120Z OK (skipped=1)
2022-11-23T02:08:16.2631139Z 
2022-11-23T02:08:16.2631246Z Generating XML reports...
2022-11-23T02:08:16.2631697Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015519.xml
2022-11-23T02:08:16.2632073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2632248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2632632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2632829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2632848Z 
2022-11-23T02:08:16.2632955Z Running tests...
2022-11-23T02:08:16.2633226Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2633522Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2633851Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-11-23T02:08:16.2633873Z 
2022-11-23T02:08:16.2634134Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2634310Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2634329Z 
2022-11-23T02:08:16.2634437Z OK (skipped=1)
2022-11-23T02:08:16.2634457Z 
2022-11-23T02:08:16.2634582Z Generating XML reports...
2022-11-23T02:08:16.2635033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015523.xml
2022-11-23T02:08:16.2635409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2635588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2635956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2636149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2636169Z 
2022-11-23T02:08:16.2636275Z Running tests...
2022-11-23T02:08:16.2636543Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2636864Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2637114Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2637857Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.671s)
2022-11-23T02:08:16.2637878Z 
2022-11-23T02:08:16.2638191Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2638312Z Ran 1 test in 1.671s
2022-11-23T02:08:16.2638331Z 
2022-11-23T02:08:16.2638439Z OK (skipped=1)
2022-11-23T02:08:16.2638458Z 
2022-11-23T02:08:16.2638564Z Generating XML reports...
2022-11-23T02:08:16.2639016Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015525.xml
2022-11-23T02:08:16.2639390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2639567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2639946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2640139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2640159Z 
2022-11-23T02:08:16.2640265Z Running tests...
2022-11-23T02:08:16.2640532Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2640830Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2641105Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2641326Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30070
2022-11-23T02:08:16.2641553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30071
2022-11-23T02:08:16.2641930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2642104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2642481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2642674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2643038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2643194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2643565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2643830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2644076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2644324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2644727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2645130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2645361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2645595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2645837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpafihwv
2022-11-23T02:08:16.2646114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpafihwv/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2646368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolj_zszs
2022-11-23T02:08:16.2646638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolj_zszs/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2647477Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T02:08:16.2647817Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T02:08:16.2648601Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T02:08:16.2648937Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T02:08:16.2649041Z ok (6.264s)
2022-11-23T02:08:16.2649061Z 
2022-11-23T02:08:16.2649325Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2649420Z Ran 1 test in 6.264s
2022-11-23T02:08:16.2649439Z 
2022-11-23T02:08:16.2649538Z OK
2022-11-23T02:08:16.2649558Z 
2022-11-23T02:08:16.2649684Z Generating XML reports...
2022-11-23T02:08:16.2650136Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015530.xml
2022-11-23T02:08:16.2650507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2650765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2651164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2651703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2651727Z 
2022-11-23T02:08:16.2651893Z Running tests...
2022-11-23T02:08:16.2652205Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2652511Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2652825Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2653629Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.609s)
2022-11-23T02:08:16.2653741Z 
2022-11-23T02:08:16.2654062Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2654217Z Ran 1 test in 1.609s
2022-11-23T02:08:16.2654237Z 
2022-11-23T02:08:16.2654420Z OK (skipped=1)
2022-11-23T02:08:16.2654439Z 
2022-11-23T02:08:16.2654653Z Generating XML reports...
2022-11-23T02:08:16.2655146Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015538.xml
2022-11-23T02:08:16.2655559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2655784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2656153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2656390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2656409Z 
2022-11-23T02:08:16.2656553Z Running tests...
2022-11-23T02:08:16.2656856Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2657212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2657569Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2658441Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2022-11-23T02:08:16.2658464Z 
2022-11-23T02:08:16.2658782Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2658933Z Ran 1 test in 1.627s
2022-11-23T02:08:16.2658956Z 
2022-11-23T02:08:16.2659101Z OK (skipped=1)
2022-11-23T02:08:16.2659120Z 
2022-11-23T02:08:16.2659227Z Generating XML reports...
2022-11-23T02:08:16.2659720Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015542.xml
2022-11-23T02:08:16.2660165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2660399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2660875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2661105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2661124Z 
2022-11-23T02:08:16.2661269Z Running tests...
2022-11-23T02:08:16.2661572Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2661873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2662210Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2662467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30287
2022-11-23T02:08:16.2662722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30288
2022-11-23T02:08:16.2663141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2663395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2663813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2664039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2664447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2664670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2665087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2665356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2665653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2665940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2666419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2666863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2667130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2667414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.2667623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2667902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.2668350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2668828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2669134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxgmesbq
2022-11-23T02:08:16.2669488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxgmesbq/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2669779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ml9obwz
2022-11-23T02:08:16.2670089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ml9obwz/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2670374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2670596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2670912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2671179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2671321Z ok (6.551s)
2022-11-23T02:08:16.2671341Z 
2022-11-23T02:08:16.2671647Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2671827Z Ran 1 test in 6.551s
2022-11-23T02:08:16.2671848Z 
2022-11-23T02:08:16.2671977Z OK
2022-11-23T02:08:16.2671996Z 
2022-11-23T02:08:16.2672171Z Generating XML reports...
2022-11-23T02:08:16.2672611Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015547.xml
2022-11-23T02:08:16.2673021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2673234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2673648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2673882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2673902Z 
2022-11-23T02:08:16.2674048Z Running tests...
2022-11-23T02:08:16.2674384Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2674749Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2675061Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2675327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30408
2022-11-23T02:08:16.2675618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30409
2022-11-23T02:08:16.2676035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2676247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2676669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2676897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2677346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2677554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2677925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2678151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2678436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2678718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2679203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2679663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2679932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2680285Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:08:16.2680555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2680810Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:08:16.2681138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7sb3e_pe
2022-11-23T02:08:16.2681445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7sb3e_pe/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2681740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxu6d4qm
2022-11-23T02:08:16.2682053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxu6d4qm/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2682322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2682595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2682896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2683169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2683432Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:08:16.2683743Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:08:16.2684060Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:08:16.2684380Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:08:16.2684649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2684919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2685247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2685548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2685811Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:08:16.2686161Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:08:16.2686477Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T02:08:16.2686793Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T02:08:16.2687062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2687331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2687604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2687868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2688036Z ok (6.968s)
2022-11-23T02:08:16.2688056Z 
2022-11-23T02:08:16.2688318Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2688468Z Ran 1 test in 6.969s
2022-11-23T02:08:16.2688488Z 
2022-11-23T02:08:16.2688627Z OK
2022-11-23T02:08:16.2688646Z 
2022-11-23T02:08:16.2688854Z Generating XML reports...
2022-11-23T02:08:16.2689363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015556.xml
2022-11-23T02:08:16.2689773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2689986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2690409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2690720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2690742Z 
2022-11-23T02:08:16.2690837Z Running tests...
2022-11-23T02:08:16.2691145Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2691717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2692050Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2692848Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2022-11-23T02:08:16.2692874Z 
2022-11-23T02:08:16.2693174Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2693325Z Ran 1 test in 1.635s
2022-11-23T02:08:16.2693345Z 
2022-11-23T02:08:16.2693502Z OK (skipped=1)
2022-11-23T02:08:16.2693522Z 
2022-11-23T02:08:16.2693722Z Generating XML reports...
2022-11-23T02:08:16.2694163Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015605.xml
2022-11-23T02:08:16.2694580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2694794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2695208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2695436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2695545Z 
2022-11-23T02:08:16.2695700Z Running tests...
2022-11-23T02:08:16.2696020Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2696409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2696758Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2696969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30561
2022-11-23T02:08:16.2697230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30562
2022-11-23T02:08:16.2697640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2697852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2698268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2698509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2698910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2699121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2699480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2699742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2700095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2700388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2700845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2701285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2701551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2702171Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:08:16.2702438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2703055Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:08:16.2703413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuef15fv
2022-11-23T02:08:16.2703674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuef15fv/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2703976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggkzwrxj
2022-11-23T02:08:16.2704280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggkzwrxj/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2704554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2704826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2705137Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T02:08:16.2705449Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T02:08:16.2705885Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T02:08:16.2706265Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T02:08:16.2706628Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T02:08:16.2706911Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T02:08:16.2707276Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T02:08:16.2707631Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T02:08:16.2707949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2708218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2708356Z ok (6.213s)
2022-11-23T02:08:16.2708375Z 
2022-11-23T02:08:16.2708736Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2708890Z Ran 1 test in 6.213s
2022-11-23T02:08:16.2708909Z 
2022-11-23T02:08:16.2709037Z OK
2022-11-23T02:08:16.2709120Z 
2022-11-23T02:08:16.2709237Z Generating XML reports...
2022-11-23T02:08:16.2709737Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015609.xml
2022-11-23T02:08:16.2710149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2710366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2710797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2711025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2711044Z 
2022-11-23T02:08:16.2711225Z Running tests...
2022-11-23T02:08:16.2711530Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2711829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2712275Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2712296Z 
2022-11-23T02:08:16.2712598Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2728153Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2728185Z 
2022-11-23T02:08:16.2728349Z OK (skipped=1)
2022-11-23T02:08:16.2728369Z 
2022-11-23T02:08:16.2728484Z Generating XML reports...
2022-11-23T02:08:16.2729001Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015618.xml
2022-11-23T02:08:16.2729386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2729567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2729962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2730158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2730178Z 
2022-11-23T02:08:16.2730288Z Running tests...
2022-11-23T02:08:16.2730559Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2731046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2731688Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2731735Z 
2022-11-23T02:08:16.2732007Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2732124Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2732144Z 
2022-11-23T02:08:16.2732255Z OK (skipped=1)
2022-11-23T02:08:16.2732274Z 
2022-11-23T02:08:16.2732405Z Generating XML reports...
2022-11-23T02:08:16.2732863Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015621.xml
2022-11-23T02:08:16.2733239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2733416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2733801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2733977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2734014Z 
2022-11-23T02:08:16.2734107Z Running tests...
2022-11-23T02:08:16.2734372Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2734686Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2735247Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2735272Z 
2022-11-23T02:08:16.2735547Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2735666Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2735685Z 
2022-11-23T02:08:16.2735795Z OK (skipped=1)
2022-11-23T02:08:16.2735813Z 
2022-11-23T02:08:16.2735938Z Generating XML reports...
2022-11-23T02:08:16.2736374Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015623.xml
2022-11-23T02:08:16.2736747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2736924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2737310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2737505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2737524Z 
2022-11-23T02:08:16.2737634Z Running tests...
2022-11-23T02:08:16.2737899Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2738221Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2738673Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2738694Z 
2022-11-23T02:08:16.2738958Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2739053Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2739072Z 
2022-11-23T02:08:16.2739181Z OK (skipped=1)
2022-11-23T02:08:16.2739204Z 
2022-11-23T02:08:16.2739329Z Generating XML reports...
2022-11-23T02:08:16.2739780Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015625.xml
2022-11-23T02:08:16.2740151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2740433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2740821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2741015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2741034Z 
2022-11-23T02:08:16.2741125Z Running tests...
2022-11-23T02:08:16.2741392Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2741703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2742155Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2742175Z 
2022-11-23T02:08:16.2742440Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2742558Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2742577Z 
2022-11-23T02:08:16.2742686Z OK (skipped=1)
2022-11-23T02:08:16.2742704Z 
2022-11-23T02:08:16.2742829Z Generating XML reports...
2022-11-23T02:08:16.2743283Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015628.xml
2022-11-23T02:08:16.2743656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2743815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2744248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2744450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2744469Z 
2022-11-23T02:08:16.2744580Z Running tests...
2022-11-23T02:08:16.2744846Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2745164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2745612Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2745632Z 
2022-11-23T02:08:16.2745895Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2746009Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2746028Z 
2022-11-23T02:08:16.2746122Z OK (skipped=1)
2022-11-23T02:08:16.2746141Z 
2022-11-23T02:08:16.2746266Z Generating XML reports...
2022-11-23T02:08:16.2746715Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015630.xml
2022-11-23T02:08:16.2747087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2747269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2747650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2747844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2747863Z 
2022-11-23T02:08:16.2747972Z Running tests...
2022-11-23T02:08:16.2748232Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2748534Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2748981Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2749001Z 
2022-11-23T02:08:16.2749329Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2749444Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2749464Z 
2022-11-23T02:08:16.2749571Z OK (skipped=1)
2022-11-23T02:08:16.2749590Z 
2022-11-23T02:08:16.2749714Z Generating XML reports...
2022-11-23T02:08:16.2750166Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015633.xml
2022-11-23T02:08:16.2750540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2750721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2751087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2751281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2751300Z 
2022-11-23T02:08:16.2751409Z Running tests...
2022-11-23T02:08:16.2751677Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2751990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2752437Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2752457Z 
2022-11-23T02:08:16.2752718Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2752879Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2752900Z 
2022-11-23T02:08:16.2753016Z OK (skipped=1)
2022-11-23T02:08:16.2753034Z 
2022-11-23T02:08:16.2753140Z Generating XML reports...
2022-11-23T02:08:16.2753596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015635.xml
2022-11-23T02:08:16.2753969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2754155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2754538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2754729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2754749Z 
2022-11-23T02:08:16.2754858Z Running tests...
2022-11-23T02:08:16.2755125Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2755444Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2755874Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2755919Z 
2022-11-23T02:08:16.2756164Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2756277Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2756296Z 
2022-11-23T02:08:16.2756407Z OK (skipped=1)
2022-11-23T02:08:16.2756425Z 
2022-11-23T02:08:16.2756549Z Generating XML reports...
2022-11-23T02:08:16.2757001Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015637.xml
2022-11-23T02:08:16.2757370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2757549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2757930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2758105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2758142Z 
2022-11-23T02:08:16.2758294Z Running tests...
2022-11-23T02:08:16.2758561Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2758886Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2759331Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2759352Z 
2022-11-23T02:08:16.2759619Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2759740Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2759758Z 
2022-11-23T02:08:16.2759867Z OK (skipped=1)
2022-11-23T02:08:16.2759885Z 
2022-11-23T02:08:16.2760011Z Generating XML reports...
2022-11-23T02:08:16.2760480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015640.xml
2022-11-23T02:08:16.2760858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2761043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2761421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2761611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2761632Z 
2022-11-23T02:08:16.2761741Z Running tests...
2022-11-23T02:08:16.2762051Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2762379Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2762771Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2762795Z 
2022-11-23T02:08:16.2763058Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2763154Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2763174Z 
2022-11-23T02:08:16.2763282Z OK (skipped=1)
2022-11-23T02:08:16.2763301Z 
2022-11-23T02:08:16.2763424Z Generating XML reports...
2022-11-23T02:08:16.2763871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015642.xml
2022-11-23T02:08:16.2764240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2764420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2764803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2764995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2765014Z 
2022-11-23T02:08:16.2765126Z Running tests...
2022-11-23T02:08:16.2765370Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2765683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2766072Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-11-23T02:08:16.2766092Z 
2022-11-23T02:08:16.2766354Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2766469Z Ran 1 test in 0.002s
2022-11-23T02:08:16.2766489Z 
2022-11-23T02:08:16.2766596Z OK (skipped=1)
2022-11-23T02:08:16.2766615Z 
2022-11-23T02:08:16.2766740Z Generating XML reports...
2022-11-23T02:08:16.2767191Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015645.xml
2022-11-23T02:08:16.2767561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2767783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2768166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2768358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2768377Z 
2022-11-23T02:08:16.2768486Z Running tests...
2022-11-23T02:08:16.2768749Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2769066Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2769340Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2770083Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.669s)
2022-11-23T02:08:16.2770107Z 
2022-11-23T02:08:16.2770370Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2770465Z Ran 1 test in 1.669s
2022-11-23T02:08:16.2770502Z 
2022-11-23T02:08:16.2770592Z OK (skipped=1)
2022-11-23T02:08:16.2770611Z 
2022-11-23T02:08:16.2770736Z Generating XML reports...
2022-11-23T02:08:16.2771238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015647.xml
2022-11-23T02:08:16.2771894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2772083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2772472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2772669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2772689Z 
2022-11-23T02:08:16.2772799Z Running tests...
2022-11-23T02:08:16.2773046Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2773359Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2773615Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2773840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31110
2022-11-23T02:08:16.2774060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31111
2022-11-23T02:08:16.2774434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2774610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2774991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2775164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2775533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2775708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2776080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2776273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2776521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2776768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2777171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2777680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2777896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2778126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2778385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_t9ajoa
2022-11-23T02:08:16.2778659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_t9ajoa/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2778912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n6z74m1
2022-11-23T02:08:16.2779180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n6z74m1/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2779288Z ok (5.837s)
2022-11-23T02:08:16.2779308Z 
2022-11-23T02:08:16.2779576Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2779690Z Ran 1 test in 5.837s
2022-11-23T02:08:16.2779709Z 
2022-11-23T02:08:16.2779786Z OK
2022-11-23T02:08:16.2779805Z 
2022-11-23T02:08:16.2779931Z Generating XML reports...
2022-11-23T02:08:16.2780382Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015651.xml
2022-11-23T02:08:16.2780819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2781008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2781394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2781588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2781611Z 
2022-11-23T02:08:16.2781722Z Running tests...
2022-11-23T02:08:16.2781968Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2782281Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2782553Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2782773Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31224
2022-11-23T02:08:16.2782992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31225
2022-11-23T02:08:16.2783364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2783540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2783918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2784112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2784462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2784638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2785008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2785197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2785447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2785693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2786095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2786558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2786793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2787005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2787268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9gstlvp3
2022-11-23T02:08:16.2787541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9gstlvp3/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2787801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzf2pxezv
2022-11-23T02:08:16.2788072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzf2pxezv/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2788311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2788547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2788958Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpzf2pxezv'>
2022-11-23T02:08:16.2789108Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:08:16.2789511Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp9gstlvp3'>
2022-11-23T02:08:16.2789678Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:08:16.2789782Z ok (6.260s)
2022-11-23T02:08:16.2789802Z 
2022-11-23T02:08:16.2790118Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2790244Z Ran 1 test in 6.260s
2022-11-23T02:08:16.2790264Z 
2022-11-23T02:08:16.2790359Z OK
2022-11-23T02:08:16.2790378Z 
2022-11-23T02:08:16.2790502Z Generating XML reports...
2022-11-23T02:08:16.2790938Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015659.xml
2022-11-23T02:08:16.2791315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2791495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2791874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2792068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2792087Z 
2022-11-23T02:08:16.2792197Z Running tests...
2022-11-23T02:08:16.2792465Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2792780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2793058Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-11-23T02:08:16.2793077Z 
2022-11-23T02:08:16.2793321Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2793435Z Ran 1 test in 0.006s
2022-11-23T02:08:16.2793454Z 
2022-11-23T02:08:16.2793561Z OK (skipped=1)
2022-11-23T02:08:16.2793580Z 
2022-11-23T02:08:16.2793703Z Generating XML reports...
2022-11-23T02:08:16.2794155Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015708.xml
2022-11-23T02:08:16.2794524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2794705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2795084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2795277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2795297Z 
2022-11-23T02:08:16.2795388Z Running tests...
2022-11-23T02:08:16.2795717Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2796032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2796298Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2796519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31376
2022-11-23T02:08:16.2796736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31377
2022-11-23T02:08:16.2797109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2797284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2797647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2797838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2798207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2798382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2798754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2798941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2799240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2799496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2799901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2800282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2800518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2800748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2801004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlkh1mq1
2022-11-23T02:08:16.2801274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlkh1mq1/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2801530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ma9_olz
2022-11-23T02:08:16.2801797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ma9_olz/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2802034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2802264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2802353Z ok (6.138s)
2022-11-23T02:08:16.2802373Z 
2022-11-23T02:08:16.2802648Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2802762Z Ran 1 test in 6.138s
2022-11-23T02:08:16.2802781Z 
2022-11-23T02:08:16.2802875Z OK
2022-11-23T02:08:16.2802894Z 
2022-11-23T02:08:16.2803020Z Generating XML reports...
2022-11-23T02:08:16.2803527Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015711.xml
2022-11-23T02:08:16.2803909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2804086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2804450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2804643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2804733Z 
2022-11-23T02:08:16.2804853Z Running tests...
2022-11-23T02:08:16.2805118Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2805432Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2805720Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2805942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31495
2022-11-23T02:08:16.2806165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31496
2022-11-23T02:08:16.2806539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2806701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2807080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2807275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2807642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2807820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2808194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2808382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2808681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2808914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2809324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2809732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2809961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2811202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2811647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.2811907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.2812320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2812711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2812953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:08:16.2813177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:08:16.2813572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.2813965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.2814225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6s7c497m
2022-11-23T02:08:16.2814501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6s7c497m/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2814760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2aiuobl
2022-11-23T02:08:16.2815030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2aiuobl/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2815235Z ok (5.253s)
2022-11-23T02:08:16.2815259Z 
2022-11-23T02:08:16.2815530Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2815627Z Ran 1 test in 5.253s
2022-11-23T02:08:16.2815646Z 
2022-11-23T02:08:16.2815740Z OK
2022-11-23T02:08:16.2815759Z 
2022-11-23T02:08:16.2815883Z Generating XML reports...
2022-11-23T02:08:16.2816337Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015719.xml
2022-11-23T02:08:16.2816715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2816894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2817274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2817467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2817490Z 
2022-11-23T02:08:16.2817582Z Running tests...
2022-11-23T02:08:16.2817850Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2818164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2818445Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2818664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31617
2022-11-23T02:08:16.2818883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31618
2022-11-23T02:08:16.2819320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2819510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2819897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2820076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2820441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2820617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2820990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2821181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2821430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2821676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2822077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2822460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2822691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2822922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2823164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.2823404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.2823804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2824199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2824437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:08:16.2824738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:08:16.2825137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.2825513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.2825772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvqrqtfx
2022-11-23T02:08:16.2826048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvqrqtfx/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2826306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiury9n7l
2022-11-23T02:08:16.2826575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiury9n7l/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2827048Z [W ProcessGroupNCCL.cpp:950] [Rank 0] Found key in store: NCCLABORTEDCOMM:20d5a7ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-11-23T02:08:16.2827573Z [W ProcessGroupNCCL.cpp:950] [Rank 1] Found key in store: NCCLABORTEDCOMM:20d5a7ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-11-23T02:08:16.2827687Z ok (15.963s)
2022-11-23T02:08:16.2827707Z 
2022-11-23T02:08:16.2827978Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2828092Z Ran 1 test in 15.964s
2022-11-23T02:08:16.2828111Z 
2022-11-23T02:08:16.2828187Z OK
2022-11-23T02:08:16.2828229Z 
2022-11-23T02:08:16.2828336Z Generating XML reports...
2022-11-23T02:08:16.2828792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015727.xml
2022-11-23T02:08:16.2829163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2829340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2829722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2829918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2829937Z 
2022-11-23T02:08:16.2830046Z Running tests...
2022-11-23T02:08:16.2830315Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2830612Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2830925Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2831146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31739
2022-11-23T02:08:16.2831363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31740
2022-11-23T02:08:16.2831738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2831915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2832298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2832490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2832854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2833071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2833447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2833635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2833886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2834131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2834534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2834933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2835165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2835378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2835638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd466kyej
2022-11-23T02:08:16.2835907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd466kyej/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2836162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpau57qdns
2022-11-23T02:08:16.2836427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpau57qdns/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2836577Z ok (6.142s)
2022-11-23T02:08:16.2836599Z 
2022-11-23T02:08:16.2836877Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2836991Z Ran 1 test in 6.142s
2022-11-23T02:08:16.2837011Z 
2022-11-23T02:08:16.2837107Z OK
2022-11-23T02:08:16.2837126Z 
2022-11-23T02:08:16.2837232Z Generating XML reports...
2022-11-23T02:08:16.2837684Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015746.xml
2022-11-23T02:08:16.2838057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2838234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2838612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2838804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2838824Z 
2022-11-23T02:08:16.2838936Z Running tests...
2022-11-23T02:08:16.2839202Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2839497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2839791Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2840014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31858
2022-11-23T02:08:16.2840231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31859
2022-11-23T02:08:16.2840608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2840782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2841159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2841352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2841716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2841873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2842245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2842495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2842741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2842986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2843390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2843796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2844027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2844255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2844493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mdvn3f9
2022-11-23T02:08:16.2844764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mdvn3f9/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2845017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwjebhcn
2022-11-23T02:08:16.2845286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwjebhcn/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2845390Z ok (6.251s)
2022-11-23T02:08:16.2845409Z 
2022-11-23T02:08:16.2845677Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2845839Z Ran 1 test in 6.252s
2022-11-23T02:08:16.2845861Z 
2022-11-23T02:08:16.2845959Z OK
2022-11-23T02:08:16.2845978Z 
2022-11-23T02:08:16.2846086Z Generating XML reports...
2022-11-23T02:08:16.2846538Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015754.xml
2022-11-23T02:08:16.2846914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2847091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2847474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2847667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2847686Z 
2022-11-23T02:08:16.2847795Z Running tests...
2022-11-23T02:08:16.2848061Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2848380Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2848621Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2848841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31977
2022-11-23T02:08:16.2849058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31978
2022-11-23T02:08:16.2849435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2849610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2849992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2850184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2850553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2850710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2851085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2851274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2851828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2852083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2852493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2852889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2853125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2853354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2853591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6l3_muh
2022-11-23T02:08:16.2853860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6l3_muh/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2854117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yd5vpfd
2022-11-23T02:08:16.2854387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yd5vpfd/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2854492Z ok (6.131s)
2022-11-23T02:08:16.2854512Z 
2022-11-23T02:08:16.2854781Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2854894Z Ran 1 test in 6.131s
2022-11-23T02:08:16.2854914Z 
2022-11-23T02:08:16.2855008Z OK
2022-11-23T02:08:16.2855027Z 
2022-11-23T02:08:16.2855212Z Generating XML reports...
2022-11-23T02:08:16.2855685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015803.xml
2022-11-23T02:08:16.2856058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2856234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2856618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2856811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2856830Z 
2022-11-23T02:08:16.2856939Z Running tests...
2022-11-23T02:08:16.2857202Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2857515Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2857763Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2857983Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32092
2022-11-23T02:08:16.2858202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32093
2022-11-23T02:08:16.2858572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2858751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2859132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2859324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2859689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2859847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2860220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2860407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2860654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2860976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2861382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2861777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2862008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2862237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2862480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7g5gyed
2022-11-23T02:08:16.2862737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueldfejy
2022-11-23T02:08:16.2863008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7g5gyed/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2863275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueldfejy/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2864110Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:08:16.2864894Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:08:16.2865138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2865375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2865478Z ok (6.108s)
2022-11-23T02:08:16.2865497Z 
2022-11-23T02:08:16.2865772Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2865885Z Ran 1 test in 6.108s
2022-11-23T02:08:16.2865904Z 
2022-11-23T02:08:16.2865998Z OK
2022-11-23T02:08:16.2866017Z 
2022-11-23T02:08:16.2866124Z Generating XML reports...
2022-11-23T02:08:16.2866578Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015812.xml
2022-11-23T02:08:16.2866952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2867128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2867507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2867698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2867717Z 
2022-11-23T02:08:16.2867826Z Running tests...
2022-11-23T02:08:16.2868094Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2868409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2868674Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2869423Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.603s)
2022-11-23T02:08:16.2869514Z 
2022-11-23T02:08:16.2869786Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2869900Z Ran 1 test in 1.603s
2022-11-23T02:08:16.2869919Z 
2022-11-23T02:08:16.2870029Z OK (skipped=1)
2022-11-23T02:08:16.2870047Z 
2022-11-23T02:08:16.2870172Z Generating XML reports...
2022-11-23T02:08:16.2870625Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015820.xml
2022-11-23T02:08:16.2870996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2871172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2871551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2871728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2871747Z 
2022-11-23T02:08:16.2871855Z Running tests...
2022-11-23T02:08:16.2872119Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2872432Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2872714Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2873546Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.599s)
2022-11-23T02:08:16.2873569Z 
2022-11-23T02:08:16.2873845Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2873961Z Ran 1 test in 1.600s
2022-11-23T02:08:16.2873981Z 
2022-11-23T02:08:16.2874089Z OK (skipped=1)
2022-11-23T02:08:16.2874108Z 
2022-11-23T02:08:16.2874214Z Generating XML reports...
2022-11-23T02:08:16.2874667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015825.xml
2022-11-23T02:08:16.2875041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2875220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2875667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2875863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2875882Z 
2022-11-23T02:08:16.2875994Z Running tests...
2022-11-23T02:08:16.2876265Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2876581Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2876841Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2877063Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32279
2022-11-23T02:08:16.2877281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32280
2022-11-23T02:08:16.2877656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2877831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2878201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2878377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2878824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2878999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2879379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2879571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2879822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2880072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2880472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2880864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2881100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2881328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2881570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeclxim_j
2022-11-23T02:08:16.2881845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeclxim_j/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2882147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuo07af6f
2022-11-23T02:08:16.2882423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuo07af6f/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2882766Z STAGE:2022-11-23 01:58:34 32279:32279 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2883104Z STAGE:2022-11-23 01:58:34 32280:32280 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2883347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2883590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2883929Z STAGE:2022-11-23 01:58:34 32279:32279 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2884242Z STAGE:2022-11-23 01:58:34 32280:32280 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2884593Z STAGE:2022-11-23 01:58:34 32279:32279 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2884944Z STAGE:2022-11-23 01:58:34 32280:32280 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2885720Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:08:16.2886499Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:08:16.2887037Z STAGE:2022-11-23 01:58:34 32279:32279 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:58:34 32280:32280 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2887106Z 
2022-11-23T02:08:16.2887660Z STAGE:2022-11-23 01:58:34 32279:32279 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:58:34 32280:32280 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2887680Z 
2022-11-23T02:08:16.2888027Z STAGE:2022-11-23 01:58:34 32280:32280 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2888376Z STAGE:2022-11-23 01:58:34 32279:32279 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2888480Z ok (6.054s)
2022-11-23T02:08:16.2888500Z 
2022-11-23T02:08:16.2888763Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2888878Z Ran 1 test in 6.055s
2022-11-23T02:08:16.2888897Z 
2022-11-23T02:08:16.2888990Z OK
2022-11-23T02:08:16.2889010Z 
2022-11-23T02:08:16.2889134Z Generating XML reports...
2022-11-23T02:08:16.2889573Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015829.xml
2022-11-23T02:08:16.2889944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2890123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2890505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2890748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2890770Z 
2022-11-23T02:08:16.2890884Z Running tests...
2022-11-23T02:08:16.2891151Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2891705Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2891968Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2892198Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32401
2022-11-23T02:08:16.2892422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32402
2022-11-23T02:08:16.2892804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2892982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2893361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2893553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2893916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2894090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2894456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2894643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2894892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2895138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2895542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2895939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2896172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2896399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2896753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25i6rzkq
2022-11-23T02:08:16.2897006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25i6rzkq/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2897263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpscpfaha_
2022-11-23T02:08:16.2897536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpscpfaha_/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2897641Z ok (5.724s)
2022-11-23T02:08:16.2897661Z 
2022-11-23T02:08:16.2897938Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2898051Z Ran 1 test in 5.725s
2022-11-23T02:08:16.2898071Z 
2022-11-23T02:08:16.2898165Z OK
2022-11-23T02:08:16.2898183Z 
2022-11-23T02:08:16.2898311Z Generating XML reports...
2022-11-23T02:08:16.2898743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015837.xml
2022-11-23T02:08:16.2899118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2899296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2899676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2899870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2899889Z 
2022-11-23T02:08:16.2899998Z Running tests...
2022-11-23T02:08:16.2900331Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2900662Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2900941Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2901673Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.616s)
2022-11-23T02:08:16.2901718Z 
2022-11-23T02:08:16.2901963Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2902077Z Ran 1 test in 1.616s
2022-11-23T02:08:16.2902096Z 
2022-11-23T02:08:16.2902207Z OK (skipped=1)
2022-11-23T02:08:16.2902227Z 
2022-11-23T02:08:16.2902353Z Generating XML reports...
2022-11-23T02:08:16.2902805Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015846.xml
2022-11-23T02:08:16.2903177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2903353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2903791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2903968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2904008Z 
2022-11-23T02:08:16.2904100Z Running tests...
2022-11-23T02:08:16.2904362Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2904677Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2904963Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2905182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32550
2022-11-23T02:08:16.2905399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32551
2022-11-23T02:08:16.2905774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2906015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2906381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2906572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2906938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2907114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2907492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2907681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2907930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2908177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2908583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2908963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2909194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2909421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2909737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpds42vltu
2022-11-23T02:08:16.2910021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpds42vltu/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2910275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qb0st2z
2022-11-23T02:08:16.2910545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qb0st2z/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2911467Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:08:16.2911584Z   warnings.warn(
2022-11-23T02:08:16.2912490Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:08:16.2912584Z   warnings.warn(
2022-11-23T02:08:16.2912827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2913057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2913161Z ok (5.663s)
2022-11-23T02:08:16.2913181Z 
2022-11-23T02:08:16.2913451Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2913564Z Ran 1 test in 5.663s
2022-11-23T02:08:16.2913584Z 
2022-11-23T02:08:16.2913677Z OK
2022-11-23T02:08:16.2913696Z 
2022-11-23T02:08:16.2913821Z Generating XML reports...
2022-11-23T02:08:16.2914257Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015850.xml
2022-11-23T02:08:16.2914632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2914812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2915193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2915447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2915466Z 
2022-11-23T02:08:16.2915578Z Running tests...
2022-11-23T02:08:16.2915842Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2916157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2916437Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2917183Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-11-23T02:08:16.2917204Z 
2022-11-23T02:08:16.2917448Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2917562Z Ran 1 test in 1.644s
2022-11-23T02:08:16.2917581Z 
2022-11-23T02:08:16.2917690Z OK (skipped=1)
2022-11-23T02:08:16.2917709Z 
2022-11-23T02:08:16.2917835Z Generating XML reports...
2022-11-23T02:08:16.2918285Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015858.xml
2022-11-23T02:08:16.2918656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2918885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2919279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2919471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2919491Z 
2022-11-23T02:08:16.2919581Z Running tests...
2022-11-23T02:08:16.2919849Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2920164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2920438Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2920660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32702
2022-11-23T02:08:16.2920878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32703
2022-11-23T02:08:16.2921254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2921430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2921791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2921985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2922357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2922530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2922903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2923091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2923343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2923589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2923992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2924370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2924667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2924898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2925160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvj28ejl9
2022-11-23T02:08:16.2925433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvj28ejl9/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2925690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgfmegul
2022-11-23T02:08:16.2925959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgfmegul/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2926304Z STAGE:2022-11-23 01:59:07 32702:32702 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2926640Z STAGE:2022-11-23 01:59:07 32703:32703 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2927249Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.2927366Z   warnings.warn(
2022-11-23T02:08:16.2927986Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:08:16.2928149Z   warnings.warn(
2022-11-23T02:08:16.2928398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2928633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:08:16.2929188Z STAGE:2022-11-23 01:59:07 32703:32703 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:07 32702:32702 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2929212Z 
2022-11-23T02:08:16.2929780Z STAGE:2022-11-23 01:59:07 32702:32702 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:59:07 32703:32703 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2929801Z 
2022-11-23T02:08:16.2930134Z STAGE:2022-11-23 01:59:07 32702:32702 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.2930473Z STAGE:2022-11-23 01:59:08 32702:32702 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.2930824Z STAGE:2022-11-23 01:59:08 32702:32702 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.2930909Z ok (6.344s)
2022-11-23T02:08:16.2930929Z 
2022-11-23T02:08:16.2931193Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2931306Z Ran 1 test in 6.344s
2022-11-23T02:08:16.2931330Z 
2022-11-23T02:08:16.2931646Z OK
2022-11-23T02:08:16.2931670Z 
2022-11-23T02:08:16.2931812Z Generating XML reports...
2022-11-23T02:08:16.2932271Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015902.xml
2022-11-23T02:08:16.2932644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2932823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2933192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2933387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2933406Z 
2022-11-23T02:08:16.2933515Z Running tests...
2022-11-23T02:08:16.2933779Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2934095Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2934487Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2934709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32824
2022-11-23T02:08:16.2934926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32825
2022-11-23T02:08:16.2935305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2935468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2935853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2936045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2936410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2936588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2936963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2937151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2937399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2937699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2938118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2938516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2938750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2938986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2939245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_1qydcz
2022-11-23T02:08:16.2939514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_1qydcz/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2939770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsflijpo
2022-11-23T02:08:16.2940043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsflijpo/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2940129Z ok (5.730s)
2022-11-23T02:08:16.2940148Z 
2022-11-23T02:08:16.2940416Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2940531Z Ran 1 test in 5.730s
2022-11-23T02:08:16.2940551Z 
2022-11-23T02:08:16.2940647Z OK
2022-11-23T02:08:16.2940666Z 
2022-11-23T02:08:16.2940790Z Generating XML reports...
2022-11-23T02:08:16.2941245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015911.xml
2022-11-23T02:08:16.2941617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2941795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2942156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2942352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2942371Z 
2022-11-23T02:08:16.2942481Z Running tests...
2022-11-23T02:08:16.2942743Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2943057Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2943330Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2943617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32939
2022-11-23T02:08:16.2943837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32940
2022-11-23T02:08:16.2944212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2944370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2944753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2944946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2945315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2945489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2945869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2946057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2946305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2946534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2946981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2947395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2947625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2947856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2948118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jwp9odo
2022-11-23T02:08:16.2948391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jwp9odo/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2948647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy87o1ox
2022-11-23T02:08:16.2948915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy87o1ox/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2949001Z ok (5.734s)
2022-11-23T02:08:16.2949021Z 
2022-11-23T02:08:16.2949297Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2949411Z Ran 1 test in 5.735s
2022-11-23T02:08:16.2949431Z 
2022-11-23T02:08:16.2949526Z OK
2022-11-23T02:08:16.2949545Z 
2022-11-23T02:08:16.2949672Z Generating XML reports...
2022-11-23T02:08:16.2950119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015919.xml
2022-11-23T02:08:16.2950496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2950671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2951050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2951222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2951242Z 
2022-11-23T02:08:16.2951353Z Running tests...
2022-11-23T02:08:16.2951623Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2951938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2952214Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2952961Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.650s)
2022-11-23T02:08:16.2953040Z 
2022-11-23T02:08:16.2953312Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2953425Z Ran 1 test in 1.650s
2022-11-23T02:08:16.2953445Z 
2022-11-23T02:08:16.2953554Z OK (skipped=1)
2022-11-23T02:08:16.2953572Z 
2022-11-23T02:08:16.2953678Z Generating XML reports...
2022-11-23T02:08:16.2954137Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015927.xml
2022-11-23T02:08:16.2954507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2954685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2955069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2955262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2955281Z 
2022-11-23T02:08:16.2955391Z Running tests...
2022-11-23T02:08:16.2955657Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2955973Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2956264Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2957016Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2022-11-23T02:08:16.2957036Z 
2022-11-23T02:08:16.2957311Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2957424Z Ran 1 test in 1.628s
2022-11-23T02:08:16.2957443Z 
2022-11-23T02:08:16.2957553Z OK (skipped=1)
2022-11-23T02:08:16.2957573Z 
2022-11-23T02:08:16.2957697Z Generating XML reports...
2022-11-23T02:08:16.2958148Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015932.xml
2022-11-23T02:08:16.2958521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2958700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2959083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2959257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2959276Z 
2022-11-23T02:08:16.2959387Z Running tests...
2022-11-23T02:08:16.2959655Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2959968Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2960261Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2961005Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.597s)
2022-11-23T02:08:16.2961026Z 
2022-11-23T02:08:16.2961289Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2961402Z Ran 1 test in 1.597s
2022-11-23T02:08:16.2961421Z 
2022-11-23T02:08:16.2961529Z OK (skipped=1)
2022-11-23T02:08:16.2961548Z 
2022-11-23T02:08:16.2961711Z Generating XML reports...
2022-11-23T02:08:16.2962167Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015936.xml
2022-11-23T02:08:16.2962539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2962717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2963096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2963291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2963311Z 
2022-11-23T02:08:16.2963422Z Running tests...
2022-11-23T02:08:16.2963686Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2964001Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2964281Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2964505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33156
2022-11-23T02:08:16.2964726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33157
2022-11-23T02:08:16.2965102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2965276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2965703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2965904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2966274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2966428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2966808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2967001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2967248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2967495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2967901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2968300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2968530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2968758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2969003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa077g_6h
2022-11-23T02:08:16.2969273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa077g_6h/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2969528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4st6p_dw
2022-11-23T02:08:16.2969796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4st6p_dw/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2969900Z ok (6.255s)
2022-11-23T02:08:16.2969920Z 
2022-11-23T02:08:16.2970192Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2970304Z Ran 1 test in 6.255s
2022-11-23T02:08:16.2970324Z 
2022-11-23T02:08:16.2970418Z OK
2022-11-23T02:08:16.2970436Z 
2022-11-23T02:08:16.2970560Z Generating XML reports...
2022-11-23T02:08:16.2970997Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015940.xml
2022-11-23T02:08:16.2971671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2971866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2972252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2972445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2972464Z 
2022-11-23T02:08:16.2972573Z Running tests...
2022-11-23T02:08:16.2972839Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2973154Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2973406Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2973628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33275
2022-11-23T02:08:16.2973852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33276
2022-11-23T02:08:16.2974221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2974398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2974773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2975044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2975432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2975608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2975960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2976157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2976405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2976650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2977055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2977451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2977682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2978063Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T02:08:16.2978319Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T02:08:16.2978536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2978912Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T02:08:16.2979166Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T02:08:16.2979426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqkmujb9
2022-11-23T02:08:16.2979705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqkmujb9/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2979960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrahnfey
2022-11-23T02:08:16.2980230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrahnfey/_remote_module_non_scriptable.py
2022-11-23T02:08:16.2980336Z ok (5.642s)
2022-11-23T02:08:16.2980355Z 
2022-11-23T02:08:16.2980604Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2980797Z Ran 1 test in 5.642s
2022-11-23T02:08:16.2980817Z 
2022-11-23T02:08:16.2980914Z OK
2022-11-23T02:08:16.2980933Z 
2022-11-23T02:08:16.2981060Z Generating XML reports...
2022-11-23T02:08:16.2981518Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015949.xml
2022-11-23T02:08:16.2981888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2982067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2982449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2982640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2982660Z 
2022-11-23T02:08:16.2982751Z Running tests...
2022-11-23T02:08:16.2983015Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2983334Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2983597Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2983818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33390
2022-11-23T02:08:16.2984038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33391
2022-11-23T02:08:16.2984459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2984645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2985011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2985206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2985576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2985753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2986129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2986321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2986566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2986817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2987218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2987598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2987833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2988076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.2988302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2988540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.2988942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2989337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2989440Z ok (4.359s)
2022-11-23T02:08:16.2989460Z 
2022-11-23T02:08:16.2989724Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2989819Z Ran 1 test in 4.359s
2022-11-23T02:08:16.2989892Z 
2022-11-23T02:08:16.2989993Z OK
2022-11-23T02:08:16.2990012Z 
2022-11-23T02:08:16.2990139Z Generating XML reports...
2022-11-23T02:08:16.2990591Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015957.xml
2022-11-23T02:08:16.2990962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2991141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2991524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2991718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2991737Z 
2022-11-23T02:08:16.2991829Z Running tests...
2022-11-23T02:08:16.2992092Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2992407Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.2992662Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.2992881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33497
2022-11-23T02:08:16.2993099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33498
2022-11-23T02:08:16.2993470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2993697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2994070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2994262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2994626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.2994804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.2995183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.2995376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.2995622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.2995866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.2996270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2996650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.2996882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.2997129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.2997353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.2997589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.2997988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2998380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.2998487Z ok (4.369s)
2022-11-23T02:08:16.2998506Z 
2022-11-23T02:08:16.2998772Z ----------------------------------------------------------------------
2022-11-23T02:08:16.2998867Z Ran 1 test in 4.369s
2022-11-23T02:08:16.2998886Z 
2022-11-23T02:08:16.2998982Z OK
2022-11-23T02:08:16.2999001Z 
2022-11-23T02:08:16.2999203Z Generating XML reports...
2022-11-23T02:08:16.2999661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020004.xml
2022-11-23T02:08:16.3000032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3000211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3000591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3000789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3000809Z 
2022-11-23T02:08:16.3000918Z Running tests...
2022-11-23T02:08:16.3001164Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3001473Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3001751Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3002504Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.602s)
2022-11-23T02:08:16.3002525Z 
2022-11-23T02:08:16.3002785Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3002900Z Ran 1 test in 1.602s
2022-11-23T02:08:16.3002968Z 
2022-11-23T02:08:16.3003086Z OK (skipped=1)
2022-11-23T02:08:16.3003105Z 
2022-11-23T02:08:16.3003231Z Generating XML reports...
2022-11-23T02:08:16.3003739Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020011.xml
2022-11-23T02:08:16.3004091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3004274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3004654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3004847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3004866Z 
2022-11-23T02:08:16.3004975Z Running tests...
2022-11-23T02:08:16.3005244Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3005560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3005841Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3006586Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.625s)
2022-11-23T02:08:16.3006610Z 
2022-11-23T02:08:16.3006874Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3006970Z Ran 1 test in 1.626s
2022-11-23T02:08:16.3006989Z 
2022-11-23T02:08:16.3007098Z OK (skipped=1)
2022-11-23T02:08:16.3007117Z 
2022-11-23T02:08:16.3007241Z Generating XML reports...
2022-11-23T02:08:16.3007696Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020015.xml
2022-11-23T02:08:16.3008071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3008247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3008625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3008883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3008903Z 
2022-11-23T02:08:16.3009013Z Running tests...
2022-11-23T02:08:16.3009259Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3009573Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3009849Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3010075Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33672
2022-11-23T02:08:16.3010293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33673
2022-11-23T02:08:16.3010665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3010840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3011216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3011389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3012000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3012177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3012552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3012824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3013084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3013330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3013734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3014141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3014355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3014586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3014689Z ok (4.340s)
2022-11-23T02:08:16.3014710Z 
2022-11-23T02:08:16.3014980Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3015095Z Ran 1 test in 4.340s
2022-11-23T02:08:16.3015114Z 
2022-11-23T02:08:16.3015209Z OK
2022-11-23T02:08:16.3015227Z 
2022-11-23T02:08:16.3015354Z Generating XML reports...
2022-11-23T02:08:16.3015808Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020019.xml
2022-11-23T02:08:16.3016165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3016346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3016723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3016913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3016933Z 
2022-11-23T02:08:16.3017042Z Running tests...
2022-11-23T02:08:16.3017311Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3017628Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3017879Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3017899Z 
2022-11-23T02:08:16.3018161Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3018372Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3018391Z 
2022-11-23T02:08:16.3018502Z OK (skipped=1)
2022-11-23T02:08:16.3018521Z 
2022-11-23T02:08:16.3018655Z Generating XML reports...
2022-11-23T02:08:16.3019113Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020026.xml
2022-11-23T02:08:16.3019485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3019662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3020044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3020236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3020256Z 
2022-11-23T02:08:16.3020366Z Running tests...
2022-11-23T02:08:16.3020611Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3020930Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3021190Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3021209Z 
2022-11-23T02:08:16.3021469Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3021584Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3021604Z 
2022-11-23T02:08:16.3021710Z OK (skipped=1)
2022-11-23T02:08:16.3021729Z 
2022-11-23T02:08:16.3021852Z Generating XML reports...
2022-11-23T02:08:16.3022350Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020028.xml
2022-11-23T02:08:16.3022712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3022892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3023280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3023470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3023490Z 
2022-11-23T02:08:16.3023599Z Running tests...
2022-11-23T02:08:16.3023865Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3024178Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3024431Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3024655Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33843
2022-11-23T02:08:16.3024855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33844
2022-11-23T02:08:16.3025228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3025409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3025781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3025955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3026333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3026525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3026907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3027079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3027325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3027571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3028040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3028434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3028667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3028897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3029240Z STAGE:2022-11-23 02:00:35 33843:33843 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3029573Z STAGE:2022-11-23 02:00:36 33844:33844 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3029889Z STAGE:2022-11-23 02:00:36 33843:33843 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3030249Z STAGE:2022-11-23 02:00:36 33843:33843 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3030584Z STAGE:2022-11-23 02:00:36 33844:33844 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3030934Z STAGE:2022-11-23 02:00:36 33844:33844 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3031261Z STAGE:2022-11-23 02:00:37 33843:33843 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3031636Z STAGE:2022-11-23 02:00:37 33844:33844 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3031986Z STAGE:2022-11-23 02:00:37 33844:33844 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3032332Z STAGE:2022-11-23 02:00:37 33844:33844 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3032660Z STAGE:2022-11-23 02:00:37 33843:33843 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3032991Z STAGE:2022-11-23 02:00:37 33843:33843 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3033098Z ok (6.562s)
2022-11-23T02:08:16.3033117Z 
2022-11-23T02:08:16.3033384Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3033496Z Ran 1 test in 6.562s
2022-11-23T02:08:16.3033516Z 
2022-11-23T02:08:16.3033610Z OK
2022-11-23T02:08:16.3033629Z 
2022-11-23T02:08:16.3033754Z Generating XML reports...
2022-11-23T02:08:16.3034212Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020031.xml
2022-11-23T02:08:16.3034583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3034760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3035126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3035324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3035343Z 
2022-11-23T02:08:16.3035454Z Running tests...
2022-11-23T02:08:16.3035718Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3036032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3036298Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3036318Z 
2022-11-23T02:08:16.3036578Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3036691Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3036710Z 
2022-11-23T02:08:16.3036800Z OK (skipped=1)
2022-11-23T02:08:16.3036839Z 
2022-11-23T02:08:16.3036945Z Generating XML reports...
2022-11-23T02:08:16.3037394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020040.xml
2022-11-23T02:08:16.3037831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3038009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3038388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3038580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3038600Z 
2022-11-23T02:08:16.3038711Z Running tests...
2022-11-23T02:08:16.3038978Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3039276Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3039538Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3039557Z 
2022-11-23T02:08:16.3039826Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3039939Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3039958Z 
2022-11-23T02:08:16.3040067Z OK (skipped=1)
2022-11-23T02:08:16.3040085Z 
2022-11-23T02:08:16.3040208Z Generating XML reports...
2022-11-23T02:08:16.3040662Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020042.xml
2022-11-23T02:08:16.3041036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3041265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3041643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3041836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3041856Z 
2022-11-23T02:08:16.3041966Z Running tests...
2022-11-23T02:08:16.3042234Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3042548Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3042801Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3043022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34032
2022-11-23T02:08:16.3043242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34033
2022-11-23T02:08:16.3043601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3043777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3044156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3044349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3044714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3044889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3045269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3045457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3045704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3045934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3046338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3046735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3047025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3047255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3047358Z ok (5.759s)
2022-11-23T02:08:16.3047378Z 
2022-11-23T02:08:16.3047647Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3047759Z Ran 1 test in 5.760s
2022-11-23T02:08:16.3047779Z 
2022-11-23T02:08:16.3047854Z OK
2022-11-23T02:08:16.3047892Z 
2022-11-23T02:08:16.3048003Z Generating XML reports...
2022-11-23T02:08:16.3048451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020045.xml
2022-11-23T02:08:16.3048819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3048994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3049377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3049566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3049585Z 
2022-11-23T02:08:16.3049694Z Running tests...
2022-11-23T02:08:16.3049955Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3050249Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3050567Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3051331Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.590s)
2022-11-23T02:08:16.3051355Z 
2022-11-23T02:08:16.3051867Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3051985Z Ran 1 test in 1.590s
2022-11-23T02:08:16.3052005Z 
2022-11-23T02:08:16.3052120Z OK (skipped=1)
2022-11-23T02:08:16.3052139Z 
2022-11-23T02:08:16.3052265Z Generating XML reports...
2022-11-23T02:08:16.3052715Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020053.xml
2022-11-23T02:08:16.3053095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3053272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3053632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3053825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3053848Z 
2022-11-23T02:08:16.3053951Z Running tests...
2022-11-23T02:08:16.3054207Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3054524Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3054769Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3054979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34184
2022-11-23T02:08:16.3055192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34185
2022-11-23T02:08:16.3055551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3055723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3056098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3056385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3056754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3056926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3057299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3057491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3057742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3057972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3058379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3058776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3059013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3059257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.3059484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3059721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.3060186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3060588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3060674Z ok (4.326s)
2022-11-23T02:08:16.3060693Z 
2022-11-23T02:08:16.3060956Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3061077Z Ran 1 test in 4.326s
2022-11-23T02:08:16.3061096Z 
2022-11-23T02:08:16.3061190Z OK
2022-11-23T02:08:16.3061209Z 
2022-11-23T02:08:16.3061334Z Generating XML reports...
2022-11-23T02:08:16.3061786Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020057.xml
2022-11-23T02:08:16.3062157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3062338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3062699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3062893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3062912Z 
2022-11-23T02:08:16.3063020Z Running tests...
2022-11-23T02:08:16.3063280Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3063598Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3063843Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3064062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34291
2022-11-23T02:08:16.3064280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34292
2022-11-23T02:08:16.3064655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3064815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3065193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3065383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3065830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3066006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3066376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3066563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3066810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3067041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3067444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3067840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3068073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3068303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3068405Z ok (5.631s)
2022-11-23T02:08:16.3068425Z 
2022-11-23T02:08:16.3068690Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3068804Z Ran 1 test in 5.631s
2022-11-23T02:08:16.3068824Z 
2022-11-23T02:08:16.3068918Z OK
2022-11-23T02:08:16.3068937Z 
2022-11-23T02:08:16.3069043Z Generating XML reports...
2022-11-23T02:08:16.3069551Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020104.xml
2022-11-23T02:08:16.3069937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3070114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3070497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3070689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3070708Z 
2022-11-23T02:08:16.3070816Z Running tests...
2022-11-23T02:08:16.3071076Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3071373Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3071620Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3071837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34406
2022-11-23T02:08:16.3072057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34407
2022-11-23T02:08:16.3072429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3072608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3072985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3073174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3073538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3073695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3074068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3074257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3074503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3074749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3075208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3075603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3075835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3076057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3076147Z ok (4.450s)
2022-11-23T02:08:16.3076166Z 
2022-11-23T02:08:16.3076434Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3076546Z Ran 1 test in 4.450s
2022-11-23T02:08:16.3076565Z 
2022-11-23T02:08:16.3076658Z OK
2022-11-23T02:08:16.3076677Z 
2022-11-23T02:08:16.3076801Z Generating XML reports...
2022-11-23T02:08:16.3077246Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020112.xml
2022-11-23T02:08:16.3077623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3077799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3078158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3078351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3078370Z 
2022-11-23T02:08:16.3078531Z Running tests...
2022-11-23T02:08:16.3078808Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3079122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3079392Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3079613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34511
2022-11-23T02:08:16.3079831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34512
2022-11-23T02:08:16.3080201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3080360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3080733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3080928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3081290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3081465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3081843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3082032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3082278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3082506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3082904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3083302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3083533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3083775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.3084000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3084302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.3084702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3085093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3085179Z ok (4.348s)
2022-11-23T02:08:16.3085198Z 
2022-11-23T02:08:16.3085470Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3085586Z Ran 1 test in 4.349s
2022-11-23T02:08:16.3085605Z 
2022-11-23T02:08:16.3085699Z OK
2022-11-23T02:08:16.3085718Z 
2022-11-23T02:08:16.3085844Z Generating XML reports...
2022-11-23T02:08:16.3086295Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020119.xml
2022-11-23T02:08:16.3086672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3086850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3087233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3087408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3087428Z 
2022-11-23T02:08:16.3087538Z Running tests...
2022-11-23T02:08:16.3087854Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3088182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3088442Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3088663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34618
2022-11-23T02:08:16.3088886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34619
2022-11-23T02:08:16.3089256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3089412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3089793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3089984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3090352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3090524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3090898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3091092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3091340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3091908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3092308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3092709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3092941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3093185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.3093411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3093743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.3094146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3094536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3094641Z ok (4.352s)
2022-11-23T02:08:16.3094661Z 
2022-11-23T02:08:16.3094908Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3095023Z Ran 1 test in 4.352s
2022-11-23T02:08:16.3095043Z 
2022-11-23T02:08:16.3095136Z OK
2022-11-23T02:08:16.3095155Z 
2022-11-23T02:08:16.3095283Z Generating XML reports...
2022-11-23T02:08:16.3095734Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020126.xml
2022-11-23T02:08:16.3096108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3096289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3096674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3096846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3096884Z 
2022-11-23T02:08:16.3096975Z Running tests...
2022-11-23T02:08:16.3097236Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3097616Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3097891Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3098111Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34725
2022-11-23T02:08:16.3098331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34726
2022-11-23T02:08:16.3098709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3098886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3099247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3099438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3099810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3099985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3100357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3100547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3100797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3101041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3101422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3101820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3102053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3102283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3102540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzqrwtge
2022-11-23T02:08:16.3102812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzqrwtge/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3103125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegxupuqr
2022-11-23T02:08:16.3103397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegxupuqr/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3103546Z ok (5.736s)
2022-11-23T02:08:16.3103567Z 
2022-11-23T02:08:16.3103822Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3103936Z Ran 1 test in 5.736s
2022-11-23T02:08:16.3103955Z 
2022-11-23T02:08:16.3104049Z OK
2022-11-23T02:08:16.3104068Z 
2022-11-23T02:08:16.3104198Z Generating XML reports...
2022-11-23T02:08:16.3104646Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020133.xml
2022-11-23T02:08:16.3105016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3105190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3105574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3105766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3105787Z 
2022-11-23T02:08:16.3105879Z Running tests...
2022-11-23T02:08:16.3106142Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3106458Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3106743Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-11-23T02:08:16.3106765Z 
2022-11-23T02:08:16.3107035Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3107150Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3107169Z 
2022-11-23T02:08:16.3107278Z OK (skipped=1)
2022-11-23T02:08:16.3107297Z 
2022-11-23T02:08:16.3107422Z Generating XML reports...
2022-11-23T02:08:16.3107857Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020141.xml
2022-11-23T02:08:16.3108231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3108406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3108788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3108987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3109006Z 
2022-11-23T02:08:16.3109114Z Running tests...
2022-11-23T02:08:16.3109379Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3109697Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3109934Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T02:08:16.3109957Z 
2022-11-23T02:08:16.3110198Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3110311Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3110330Z 
2022-11-23T02:08:16.3110438Z OK (skipped=1)
2022-11-23T02:08:16.3110456Z 
2022-11-23T02:08:16.3110581Z Generating XML reports...
2022-11-23T02:08:16.3111033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020144.xml
2022-11-23T02:08:16.3111407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3111584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3111963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3112154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3112227Z 
2022-11-23T02:08:16.3112323Z Running tests...
2022-11-23T02:08:16.3112589Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3112903Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3113170Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T02:08:16.3113190Z 
2022-11-23T02:08:16.3113449Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3113560Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3113583Z 
2022-11-23T02:08:16.3113689Z OK (skipped=1)
2022-11-23T02:08:16.3113707Z 
2022-11-23T02:08:16.3113830Z Generating XML reports...
2022-11-23T02:08:16.3114257Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020146.xml
2022-11-23T02:08:16.3114625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3114800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3115179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3115368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3115387Z 
2022-11-23T02:08:16.3115488Z Running tests...
2022-11-23T02:08:16.3115750Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3116113Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3116379Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-11-23T02:08:16.3116399Z 
2022-11-23T02:08:16.3116641Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3116753Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3116776Z 
2022-11-23T02:08:16.3116882Z OK (skipped=1)
2022-11-23T02:08:16.3116901Z 
2022-11-23T02:08:16.3117028Z Generating XML reports...
2022-11-23T02:08:16.3117476Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020149.xml
2022-11-23T02:08:16.3117848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3118019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3118398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3118590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3118609Z 
2022-11-23T02:08:16.3118701Z Running tests...
2022-11-23T02:08:16.3118965Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3119278Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3119562Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3119783Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34975
2022-11-23T02:08:16.3120000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34976
2022-11-23T02:08:16.3120371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3120545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3120905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3121096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3121463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3121693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3122066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3122246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3122493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3122744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3123141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3123522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3123754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3124001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.3124225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3124461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.3124862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3125305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3125554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:08:16.3125794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:08:16.3126168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.3126571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.3126801Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:08:16.3126907Z ok (21.567s)
2022-11-23T02:08:16.3126926Z 
2022-11-23T02:08:16.3127190Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3127304Z Ran 1 test in 21.567s
2022-11-23T02:08:16.3127327Z 
2022-11-23T02:08:16.3127420Z OK
2022-11-23T02:08:16.3127439Z 
2022-11-23T02:08:16.3127564Z Generating XML reports...
2022-11-23T02:08:16.3128014Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020151.xml
2022-11-23T02:08:16.3128369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3128549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3128929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3129121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3129141Z 
2022-11-23T02:08:16.3129250Z Running tests...
2022-11-23T02:08:16.3129517Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3129834Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3130134Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3130337Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35098
2022-11-23T02:08:16.3130556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35099
2022-11-23T02:08:16.3131008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3131185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3131804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3132005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3132379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3132556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3132928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3133098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3133347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3133594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3133996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3134391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3134708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3134964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.3135192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3135429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.3135817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3136207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3136449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:08:16.3136690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:08:16.3137090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.3137478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.3137712Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:08:16.3137946Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:08:16.3138049Z ok (21.381s)
2022-11-23T02:08:16.3138069Z 
2022-11-23T02:08:16.3138321Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3138435Z Ran 1 test in 21.382s
2022-11-23T02:08:16.3138454Z 
2022-11-23T02:08:16.3138549Z OK
2022-11-23T02:08:16.3138568Z 
2022-11-23T02:08:16.3138693Z Generating XML reports...
2022-11-23T02:08:16.3139146Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020215.xml
2022-11-23T02:08:16.3139522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3139699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3140081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3140339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3140377Z 
2022-11-23T02:08:16.3140469Z Running tests...
2022-11-23T02:08:16.3140734Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3141051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3141467Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.3141488Z 
2022-11-23T02:08:16.3141750Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3141863Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3141882Z 
2022-11-23T02:08:16.3141992Z OK (skipped=1)
2022-11-23T02:08:16.3142011Z 
2022-11-23T02:08:16.3142135Z Generating XML reports...
2022-11-23T02:08:16.3142567Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020239.xml
2022-11-23T02:08:16.3142946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3143121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3143505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3143695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3143714Z 
2022-11-23T02:08:16.3143825Z Running tests...
2022-11-23T02:08:16.3144136Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3144461Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3144854Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.3144874Z 
2022-11-23T02:08:16.3145120Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3145232Z Ran 1 test in 0.003s
2022-11-23T02:08:16.3145252Z 
2022-11-23T02:08:16.3145360Z OK (skipped=1)
2022-11-23T02:08:16.3145379Z 
2022-11-23T02:08:16.3145507Z Generating XML reports...
2022-11-23T02:08:16.3145953Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020241.xml
2022-11-23T02:08:16.3146323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3146501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3146881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3147070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3147090Z 
2022-11-23T02:08:16.3147180Z Running tests...
2022-11-23T02:08:16.3147448Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3147763Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3148184Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.3148204Z 
2022-11-23T02:08:16.3148467Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3148578Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3148597Z 
2022-11-23T02:08:16.3148705Z OK (skipped=1)
2022-11-23T02:08:16.3148727Z 
2022-11-23T02:08:16.3148851Z Generating XML reports...
2022-11-23T02:08:16.3149277Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020244.xml
2022-11-23T02:08:16.3149647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3149886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3150267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3150459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3150478Z 
2022-11-23T02:08:16.3150586Z Running tests...
2022-11-23T02:08:16.3150852Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3151166Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3151580Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.3151601Z 
2022-11-23T02:08:16.3151843Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3151956Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3151975Z 
2022-11-23T02:08:16.3152085Z OK (skipped=1)
2022-11-23T02:08:16.3152104Z 
2022-11-23T02:08:16.3152226Z Generating XML reports...
2022-11-23T02:08:16.3152678Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020246.xml
2022-11-23T02:08:16.3153048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3153224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3153651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3153852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3153872Z 
2022-11-23T02:08:16.3153962Z Running tests...
2022-11-23T02:08:16.3154228Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3154539Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3154955Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.3154975Z 
2022-11-23T02:08:16.3155237Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3155349Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3155368Z 
2022-11-23T02:08:16.3155476Z OK (skipped=1)
2022-11-23T02:08:16.3155494Z 
2022-11-23T02:08:16.3155618Z Generating XML reports...
2022-11-23T02:08:16.3156067Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020249.xml
2022-11-23T02:08:16.3156420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3156595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3156975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3157170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3157190Z 
2022-11-23T02:08:16.3157298Z Running tests...
2022-11-23T02:08:16.3157559Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3157873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3158148Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3158355Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35386
2022-11-23T02:08:16.3158575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35387
2022-11-23T02:08:16.3158947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3159124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3159570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3159765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3160131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3160306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3160679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3160850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3161098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3161343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3161747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3162145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3162376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3162604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3162707Z ok (5.166s)
2022-11-23T02:08:16.3162727Z 
2022-11-23T02:08:16.3163081Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3163184Z Ran 1 test in 5.166s
2022-11-23T02:08:16.3163203Z 
2022-11-23T02:08:16.3163298Z OK
2022-11-23T02:08:16.3163317Z 
2022-11-23T02:08:16.3163443Z Generating XML reports...
2022-11-23T02:08:16.3163896Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020251.xml
2022-11-23T02:08:16.3164270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3164446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3164821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3165013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3165032Z 
2022-11-23T02:08:16.3165123Z Running tests...
2022-11-23T02:08:16.3165386Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3165698Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3165976Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3166197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35500
2022-11-23T02:08:16.3166419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35501
2022-11-23T02:08:16.3166790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3166965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3167349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3167527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3167892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3168067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3168444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3168693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3168941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3169184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3169589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3169971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3170201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3170432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3170535Z ok (5.167s)
2022-11-23T02:08:16.3170555Z 
2022-11-23T02:08:16.3170824Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3170936Z Ran 1 test in 5.167s
2022-11-23T02:08:16.3170956Z 
2022-11-23T02:08:16.3171051Z OK
2022-11-23T02:08:16.3171070Z 
2022-11-23T02:08:16.3171198Z Generating XML reports...
2022-11-23T02:08:16.3171889Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020259.xml
2022-11-23T02:08:16.3172258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3172516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3172914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3173105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3173125Z 
2022-11-23T02:08:16.3173232Z Running tests...
2022-11-23T02:08:16.3173500Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3173815Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3174091Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3174292Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35614
2022-11-23T02:08:16.3174514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35615
2022-11-23T02:08:16.3174890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3175065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3175444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3175697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3176070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3176246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3176618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3176790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3177035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3177287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3177689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3178087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3178407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3178637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3178742Z ok (5.627s)
2022-11-23T02:08:16.3178763Z 
2022-11-23T02:08:16.3179031Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3179125Z Ran 1 test in 5.627s
2022-11-23T02:08:16.3179144Z 
2022-11-23T02:08:16.3179238Z OK
2022-11-23T02:08:16.3179257Z 
2022-11-23T02:08:16.3179386Z Generating XML reports...
2022-11-23T02:08:16.3179839Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020306.xml
2022-11-23T02:08:16.3180214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3180389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3180771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3180960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3180980Z 
2022-11-23T02:08:16.3181072Z Running tests...
2022-11-23T02:08:16.3181335Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3181649Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3181975Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3182208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35729
2022-11-23T02:08:16.3182428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35730
2022-11-23T02:08:16.3182804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3182983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3183365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3183538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3183906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3184081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3184462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3184650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3184897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3185146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3185547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3185924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3186153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3186385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3186490Z ok (5.257s)
2022-11-23T02:08:16.3186509Z 
2022-11-23T02:08:16.3186774Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3186887Z Ran 1 test in 5.257s
2022-11-23T02:08:16.3186906Z 
2022-11-23T02:08:16.3187000Z OK
2022-11-23T02:08:16.3187019Z 
2022-11-23T02:08:16.3187142Z Generating XML reports...
2022-11-23T02:08:16.3187661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020315.xml
2022-11-23T02:08:16.3188016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3188193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3188575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3188771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3188790Z 
2022-11-23T02:08:16.3188901Z Running tests...
2022-11-23T02:08:16.3189164Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3189477Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3189762Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3190516Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2022-11-23T02:08:16.3190536Z 
2022-11-23T02:08:16.3190799Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3190895Z Ran 1 test in 1.633s
2022-11-23T02:08:16.3190914Z 
2022-11-23T02:08:16.3191075Z OK (skipped=1)
2022-11-23T02:08:16.3191096Z 
2022-11-23T02:08:16.3191227Z Generating XML reports...
2022-11-23T02:08:16.3191680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020322.xml
2022-11-23T02:08:16.3192053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3192233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3192612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3192803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3192823Z 
2022-11-23T02:08:16.3192932Z Running tests...
2022-11-23T02:08:16.3193180Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3193500Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3193755Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:08:16.3193774Z 
2022-11-23T02:08:16.3194036Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3194148Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3194168Z 
2022-11-23T02:08:16.3194280Z OK (skipped=1)
2022-11-23T02:08:16.3194298Z 
2022-11-23T02:08:16.3194422Z Generating XML reports...
2022-11-23T02:08:16.3194875Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020327.xml
2022-11-23T02:08:16.3195227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3195405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3195791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3195983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3196002Z 
2022-11-23T02:08:16.3196111Z Running tests...
2022-11-23T02:08:16.3196376Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3196688Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3197039Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:08:16.3197058Z 
2022-11-23T02:08:16.3197321Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3197416Z Ran 1 test in 0.003s
2022-11-23T02:08:16.3197455Z 
2022-11-23T02:08:16.3197547Z OK (skipped=1)
2022-11-23T02:08:16.3197565Z 
2022-11-23T02:08:16.3197687Z Generating XML reports...
2022-11-23T02:08:16.3198144Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020329.xml
2022-11-23T02:08:16.3198515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3198691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3199072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3199267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3199287Z 
2022-11-23T02:08:16.3199394Z Running tests...
2022-11-23T02:08:16.3199637Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3199951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3200261Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:08:16.3200281Z 
2022-11-23T02:08:16.3200595Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3200714Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3200733Z 
2022-11-23T02:08:16.3200842Z OK (skipped=1)
2022-11-23T02:08:16.3200861Z 
2022-11-23T02:08:16.3200985Z Generating XML reports...
2022-11-23T02:08:16.3201437Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020331.xml
2022-11-23T02:08:16.3201812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3201970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3202349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3202540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3202559Z 
2022-11-23T02:08:16.3202673Z Running tests...
2022-11-23T02:08:16.3202935Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3203250Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3203604Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3203834Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35976
2022-11-23T02:08:16.3204034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35977
2022-11-23T02:08:16.3204409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3204584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3204964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3205159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3205527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3205701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3206074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3206326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3206559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3206965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3207211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3207614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3207844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3208074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3208178Z ok (4.135s)
2022-11-23T02:08:16.3208201Z 
2022-11-23T02:08:16.3208467Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3208581Z Ran 1 test in 4.135s
2022-11-23T02:08:16.3208600Z 
2022-11-23T02:08:16.3208677Z OK
2022-11-23T02:08:16.3208695Z 
2022-11-23T02:08:16.3208821Z Generating XML reports...
2022-11-23T02:08:16.3209274Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020334.xml
2022-11-23T02:08:16.3209644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3209870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3210260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3210452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3210472Z 
2022-11-23T02:08:16.3210582Z Running tests...
2022-11-23T02:08:16.3210829Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3211143Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3211669Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3211905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36081
2022-11-23T02:08:16.3212122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36082
2022-11-23T02:08:16.3212509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3212685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3213067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3213263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3213613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3213790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3214161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3214349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3214602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3214849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3215255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3215655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3215984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3216195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3216299Z ok (4.241s)
2022-11-23T02:08:16.3216319Z 
2022-11-23T02:08:16.3216590Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3216701Z Ran 1 test in 4.241s
2022-11-23T02:08:16.3216721Z 
2022-11-23T02:08:16.3216815Z OK
2022-11-23T02:08:16.3216838Z 
2022-11-23T02:08:16.3216964Z Generating XML reports...
2022-11-23T02:08:16.3217418Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020340.xml
2022-11-23T02:08:16.3217792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3217955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3218339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3218530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3218549Z 
2022-11-23T02:08:16.3218659Z Running tests...
2022-11-23T02:08:16.3218922Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3219234Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3219579Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:08:16.3219601Z 
2022-11-23T02:08:16.3219872Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3219987Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3220006Z 
2022-11-23T02:08:16.3220096Z OK (skipped=1)
2022-11-23T02:08:16.3220119Z 
2022-11-23T02:08:16.3220243Z Generating XML reports...
2022-11-23T02:08:16.3220694Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020347.xml
2022-11-23T02:08:16.3221065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3221242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3221626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3221817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3221836Z 
2022-11-23T02:08:16.3221945Z Running tests...
2022-11-23T02:08:16.3222190Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3222505Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3222809Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:08:16.3222829Z 
2022-11-23T02:08:16.3223088Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3223199Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3223218Z 
2022-11-23T02:08:16.3223327Z OK (skipped=1)
2022-11-23T02:08:16.3223345Z 
2022-11-23T02:08:16.3223469Z Generating XML reports...
2022-11-23T02:08:16.3223918Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020350.xml
2022-11-23T02:08:16.3224290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3224449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3224830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3225081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3225100Z 
2022-11-23T02:08:16.3225211Z Running tests...
2022-11-23T02:08:16.3225475Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3225789Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3226069Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3226820Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.651s)
2022-11-23T02:08:16.3226841Z 
2022-11-23T02:08:16.3227104Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3227218Z Ran 1 test in 1.651s
2022-11-23T02:08:16.3227237Z 
2022-11-23T02:08:16.3227327Z OK (skipped=1)
2022-11-23T02:08:16.3227346Z 
2022-11-23T02:08:16.3227470Z Generating XML reports...
2022-11-23T02:08:16.3227915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020352.xml
2022-11-23T02:08:16.3228286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3228461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3228896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3229095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3229115Z 
2022-11-23T02:08:16.3229224Z Running tests...
2022-11-23T02:08:16.3229470Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3229788Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3230070Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3230290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36286
2022-11-23T02:08:16.3230508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36287
2022-11-23T02:08:16.3230885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3231061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3231439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3231631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3231980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3232154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3232522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3232713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3232962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3233212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3233614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3234010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3234304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3234516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3234777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjc54hgv7
2022-11-23T02:08:16.3235048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjc54hgv7/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3235307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0idr30e
2022-11-23T02:08:16.3235577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0idr30e/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3235684Z ok (6.110s)
2022-11-23T02:08:16.3235703Z 
2022-11-23T02:08:16.3235973Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3236086Z Ran 1 test in 6.111s
2022-11-23T02:08:16.3236105Z 
2022-11-23T02:08:16.3236184Z OK
2022-11-23T02:08:16.3236223Z 
2022-11-23T02:08:16.3236329Z Generating XML reports...
2022-11-23T02:08:16.3236784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020356.xml
2022-11-23T02:08:16.3237160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3237339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3237770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3237971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3237990Z 
2022-11-23T02:08:16.3238099Z Running tests...
2022-11-23T02:08:16.3238365Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3238661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3238939Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3239161Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36405
2022-11-23T02:08:16.3239380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36406
2022-11-23T02:08:16.3239751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3239925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3240308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3240498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3240842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3241019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3241398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3241588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3241836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3242084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3242488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3242883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3243113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3243384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3243494Z ok (5.620s)
2022-11-23T02:08:16.3243513Z 
2022-11-23T02:08:16.3243783Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3243896Z Ran 1 test in 5.620s
2022-11-23T02:08:16.3243915Z 
2022-11-23T02:08:16.3244010Z OK
2022-11-23T02:08:16.3244028Z 
2022-11-23T02:08:16.3244153Z Generating XML reports...
2022-11-23T02:08:16.3244606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020405.xml
2022-11-23T02:08:16.3244983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3245160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3245522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3245718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3245737Z 
2022-11-23T02:08:16.3245844Z Running tests...
2022-11-23T02:08:16.3246108Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3246422Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3246709Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3246978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36520
2022-11-23T02:08:16.3247206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36521
2022-11-23T02:08:16.3247562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3247739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3248120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3248312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3248675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3248850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3249223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3249412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3249661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3249887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3250290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3250690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3250925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3251153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3251256Z ok (5.745s)
2022-11-23T02:08:16.3251275Z 
2022-11-23T02:08:16.3251773Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3251894Z Ran 1 test in 5.745s
2022-11-23T02:08:16.3251915Z 
2022-11-23T02:08:16.3251990Z OK
2022-11-23T02:08:16.3252029Z 
2022-11-23T02:08:16.3252137Z Generating XML reports...
2022-11-23T02:08:16.3252594Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020413.xml
2022-11-23T02:08:16.3253065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3253243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3253623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3253816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3253835Z 
2022-11-23T02:08:16.3253947Z Running tests...
2022-11-23T02:08:16.3254216Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3254514Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3254796Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3255542Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2022-11-23T02:08:16.3255566Z 
2022-11-23T02:08:16.3255825Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3255941Z Ran 1 test in 1.635s
2022-11-23T02:08:16.3255961Z 
2022-11-23T02:08:16.3256070Z OK (skipped=1)
2022-11-23T02:08:16.3256088Z 
2022-11-23T02:08:16.3256214Z Generating XML reports...
2022-11-23T02:08:16.3256735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020421.xml
2022-11-23T02:08:16.3257123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3257302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3257666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3257862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3257881Z 
2022-11-23T02:08:16.3257990Z Running tests...
2022-11-23T02:08:16.3258254Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3258567Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3258869Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3259616Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.600s)
2022-11-23T02:08:16.3259637Z 
2022-11-23T02:08:16.3259899Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3260013Z Ran 1 test in 1.600s
2022-11-23T02:08:16.3260032Z 
2022-11-23T02:08:16.3260123Z OK (skipped=1)
2022-11-23T02:08:16.3260160Z 
2022-11-23T02:08:16.3260266Z Generating XML reports...
2022-11-23T02:08:16.3260715Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020425.xml
2022-11-23T02:08:16.3261088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3261268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3261652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3261843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3261863Z 
2022-11-23T02:08:16.3261977Z Running tests...
2022-11-23T02:08:16.3262318Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3262616Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3262929Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3263151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36703
2022-11-23T02:08:16.3263370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36704
2022-11-23T02:08:16.3263748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3263925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3264305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3264497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3264854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3265029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3265402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3265591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3265886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3266141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3266545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3266942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3267177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3267388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3267541Z skip: Need at least 4 CUDA devices (4.256s)
2022-11-23T02:08:16.3267562Z 
2022-11-23T02:08:16.3267828Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3267941Z Ran 1 test in 4.256s
2022-11-23T02:08:16.3267960Z 
2022-11-23T02:08:16.3268071Z OK (skipped=1)
2022-11-23T02:08:16.3268090Z 
2022-11-23T02:08:16.3268214Z Generating XML reports...
2022-11-23T02:08:16.3268665Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020430.xml
2022-11-23T02:08:16.3269036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3269214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3269576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3269766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3269785Z 
2022-11-23T02:08:16.3269894Z Running tests...
2022-11-23T02:08:16.3270160Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3270480Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3270806Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3271026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36808
2022-11-23T02:08:16.3271242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36809
2022-11-23T02:08:16.3271684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3271841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3272219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3272409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3272774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3272949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3273330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3273520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3273771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3273998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3274398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3274795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3275077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3275313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3275465Z skip: Need at least 4 CUDA devices (4.248s)
2022-11-23T02:08:16.3275485Z 
2022-11-23T02:08:16.3275754Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3275870Z Ran 1 test in 4.249s
2022-11-23T02:08:16.3275889Z 
2022-11-23T02:08:16.3275996Z OK (skipped=1)
2022-11-23T02:08:16.3276015Z 
2022-11-23T02:08:16.3276122Z Generating XML reports...
2022-11-23T02:08:16.3276574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020436.xml
2022-11-23T02:08:16.3276945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3277120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3277503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3277694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3277713Z 
2022-11-23T02:08:16.3277823Z Running tests...
2022-11-23T02:08:16.3278090Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3278388Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3278678Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3279423Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.643s)
2022-11-23T02:08:16.3279444Z 
2022-11-23T02:08:16.3279712Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3279824Z Ran 1 test in 1.644s
2022-11-23T02:08:16.3279844Z 
2022-11-23T02:08:16.3279957Z OK (skipped=1)
2022-11-23T02:08:16.3279975Z 
2022-11-23T02:08:16.3280099Z Generating XML reports...
2022-11-23T02:08:16.3280545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020443.xml
2022-11-23T02:08:16.3280979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3281156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3281519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3281710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3281729Z 
2022-11-23T02:08:16.3281840Z Running tests...
2022-11-23T02:08:16.3282104Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3282420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3282689Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3282709Z 
2022-11-23T02:08:16.3282972Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3283085Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3283105Z 
2022-11-23T02:08:16.3283211Z OK (skipped=1)
2022-11-23T02:08:16.3283230Z 
2022-11-23T02:08:16.3283336Z Generating XML reports...
2022-11-23T02:08:16.3283785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020447.xml
2022-11-23T02:08:16.3284153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3284377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3284769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3284962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3284981Z 
2022-11-23T02:08:16.3285094Z Running tests...
2022-11-23T02:08:16.3285360Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3285655Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3285924Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3285943Z 
2022-11-23T02:08:16.3286203Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3286314Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3286334Z 
2022-11-23T02:08:16.3286442Z OK (skipped=1)
2022-11-23T02:08:16.3286464Z 
2022-11-23T02:08:16.3286588Z Generating XML reports...
2022-11-23T02:08:16.3287032Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020450.xml
2022-11-23T02:08:16.3287402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3287581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3287939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3288130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3288149Z 
2022-11-23T02:08:16.3288256Z Running tests...
2022-11-23T02:08:16.3288518Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3288828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3289106Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3289126Z 
2022-11-23T02:08:16.3289387Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3289499Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3289518Z 
2022-11-23T02:08:16.3289627Z OK (skipped=1)
2022-11-23T02:08:16.3289700Z 
2022-11-23T02:08:16.3289809Z Generating XML reports...
2022-11-23T02:08:16.3290258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020452.xml
2022-11-23T02:08:16.3290631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3290808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3291191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3291385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3291404Z 
2022-11-23T02:08:16.3291747Z Running tests...
2022-11-23T02:08:16.3292026Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3292341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3292600Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3292619Z 
2022-11-23T02:08:16.3292882Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3292994Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3293013Z 
2022-11-23T02:08:16.3293122Z OK (skipped=1)
2022-11-23T02:08:16.3293141Z 
2022-11-23T02:08:16.3293265Z Generating XML reports...
2022-11-23T02:08:16.3293791Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020455.xml
2022-11-23T02:08:16.3294179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3294355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3294732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3294912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3294932Z 
2022-11-23T02:08:16.3295041Z Running tests...
2022-11-23T02:08:16.3295302Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3295618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3295883Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3295903Z 
2022-11-23T02:08:16.3296163Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3296276Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3296295Z 
2022-11-23T02:08:16.3296402Z OK (skipped=1)
2022-11-23T02:08:16.3296420Z 
2022-11-23T02:08:16.3296526Z Generating XML reports...
2022-11-23T02:08:16.3296975Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020457.xml
2022-11-23T02:08:16.3297344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3297518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3297896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3298086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3298105Z 
2022-11-23T02:08:16.3298216Z Running tests...
2022-11-23T02:08:16.3298484Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3298800Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3299048Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3299068Z 
2022-11-23T02:08:16.3299323Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3299519Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3299539Z 
2022-11-23T02:08:16.3299650Z OK (skipped=1)
2022-11-23T02:08:16.3299669Z 
2022-11-23T02:08:16.3299795Z Generating XML reports...
2022-11-23T02:08:16.3300250Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020459.xml
2022-11-23T02:08:16.3300622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3300802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3301185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3301359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3301379Z 
2022-11-23T02:08:16.3301490Z Running tests...
2022-11-23T02:08:16.3301752Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3302068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3302340Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3302359Z 
2022-11-23T02:08:16.3302618Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3302729Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3302749Z 
2022-11-23T02:08:16.3302856Z OK (skipped=1)
2022-11-23T02:08:16.3302876Z 
2022-11-23T02:08:16.3303051Z Generating XML reports...
2022-11-23T02:08:16.3303491Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020502.xml
2022-11-23T02:08:16.3303916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3304092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3304475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3304668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3304688Z 
2022-11-23T02:08:16.3304802Z Running tests...
2022-11-23T02:08:16.3305066Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3305379Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3305625Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3305664Z 
2022-11-23T02:08:16.3305909Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3306019Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3306039Z 
2022-11-23T02:08:16.3306151Z OK (skipped=1)
2022-11-23T02:08:16.3306170Z 
2022-11-23T02:08:16.3306300Z Generating XML reports...
2022-11-23T02:08:16.3306747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020504.xml
2022-11-23T02:08:16.3307119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3307294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3307672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3307849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3307869Z 
2022-11-23T02:08:16.3307981Z Running tests...
2022-11-23T02:08:16.3308243Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3308555Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3308809Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3308919Z 
2022-11-23T02:08:16.3309191Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3309304Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3309323Z 
2022-11-23T02:08:16.3309430Z OK (skipped=1)
2022-11-23T02:08:16.3309449Z 
2022-11-23T02:08:16.3309577Z Generating XML reports...
2022-11-23T02:08:16.3310004Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020507.xml
2022-11-23T02:08:16.3310377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3310552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3310931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3311123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3311145Z 
2022-11-23T02:08:16.3311255Z Running tests...
2022-11-23T02:08:16.3311517Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3311828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3312066Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3312103Z 
2022-11-23T02:08:16.3312345Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3312512Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3312533Z 
2022-11-23T02:08:16.3312648Z OK (skipped=1)
2022-11-23T02:08:16.3312667Z 
2022-11-23T02:08:16.3312792Z Generating XML reports...
2022-11-23T02:08:16.3313244Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020509.xml
2022-11-23T02:08:16.3313617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3313797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3314176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3314350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3314388Z 
2022-11-23T02:08:16.3314479Z Running tests...
2022-11-23T02:08:16.3314743Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3315060Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3315320Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3315543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37277
2022-11-23T02:08:16.3315761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37278
2022-11-23T02:08:16.3316140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3316298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3316678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3316868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3317236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3317409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3317783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3317974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3318295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3318542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3318930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3319327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3319563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3319795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3320135Z STAGE:2022-11-23 02:05:16 37277:37277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3320895Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:08:16.3321014Z   warnings.warn(
2022-11-23T02:08:16.3321345Z STAGE:2022-11-23 02:05:16 37278:37278 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3322159Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:08:16.3322283Z   warnings.warn(
2022-11-23T02:08:16.3322608Z STAGE:2022-11-23 02:05:16 37277:37277 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3322943Z STAGE:2022-11-23 02:05:16 37278:37278 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3323296Z STAGE:2022-11-23 02:05:16 37277:37277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3323647Z STAGE:2022-11-23 02:05:16 37278:37278 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3323976Z STAGE:2022-11-23 02:05:16 37277:37277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3324304Z STAGE:2022-11-23 02:05:16 37277:37277 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3324637Z STAGE:2022-11-23 02:05:16 37278:37278 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3324988Z STAGE:2022-11-23 02:05:16 37277:37277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3325322Z STAGE:2022-11-23 02:05:16 37278:37278 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3325651Z STAGE:2022-11-23 02:05:16 37278:37278 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3325760Z ok (5.680s)
2022-11-23T02:08:16.3325779Z 
2022-11-23T02:08:16.3326044Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3326159Z Ran 1 test in 5.681s
2022-11-23T02:08:16.3326179Z 
2022-11-23T02:08:16.3326273Z OK
2022-11-23T02:08:16.3326292Z 
2022-11-23T02:08:16.3326417Z Generating XML reports...
2022-11-23T02:08:16.3326870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020511.xml
2022-11-23T02:08:16.3327247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3327409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3327792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3327983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3328067Z 
2022-11-23T02:08:16.3328184Z Running tests...
2022-11-23T02:08:16.3328451Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3328767Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3329034Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3329053Z 
2022-11-23T02:08:16.3329316Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3329431Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3329451Z 
2022-11-23T02:08:16.3329540Z OK (skipped=1)
2022-11-23T02:08:16.3329559Z 
2022-11-23T02:08:16.3329683Z Generating XML reports...
2022-11-23T02:08:16.3330133Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020520.xml
2022-11-23T02:08:16.3330506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3330684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3331064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3331257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3331277Z 
2022-11-23T02:08:16.3331386Z Running tests...
2022-11-23T02:08:16.3331853Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3332235Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3332522Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3332744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37428
2022-11-23T02:08:16.3332964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37429
2022-11-23T02:08:16.3333342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3333518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3333898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3334091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3334439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3334613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3334988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3335181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3335437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3335683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3336084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3336486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3336720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3336933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3337270Z STAGE:2022-11-23 02:05:27 37429:37429 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3337768Z STAGE:2022-11-23 02:05:27 37428:37428 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3338254Z STAGE:2022-11-23 02:05:28 37428:37428 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3338591Z STAGE:2022-11-23 02:05:28 37429:37429 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3338940Z STAGE:2022-11-23 02:05:28 37428:37428 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3339287Z STAGE:2022-11-23 02:05:28 37429:37429 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3339729Z STAGE:2022-11-23 02:05:28 37429:37429 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3340073Z STAGE:2022-11-23 02:05:28 37429:37429 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3340401Z STAGE:2022-11-23 02:05:28 37429:37429 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3340729Z STAGE:2022-11-23 02:05:28 37428:37428 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3341060Z STAGE:2022-11-23 02:05:28 37428:37428 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3341403Z STAGE:2022-11-23 02:05:28 37428:37428 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3341509Z ok (6.469s)
2022-11-23T02:08:16.3341532Z 
2022-11-23T02:08:16.3341799Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3341912Z Ran 1 test in 6.469s
2022-11-23T02:08:16.3341932Z 
2022-11-23T02:08:16.3342088Z OK
2022-11-23T02:08:16.3342109Z 
2022-11-23T02:08:16.3342222Z Generating XML reports...
2022-11-23T02:08:16.3342681Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020522.xml
2022-11-23T02:08:16.3343053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3343238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3343619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3343816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3343835Z 
2022-11-23T02:08:16.3343945Z Running tests...
2022-11-23T02:08:16.3344207Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3344527Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3344777Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3345001Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37547
2022-11-23T02:08:16.3345219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37548
2022-11-23T02:08:16.3345595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3345771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3346155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3346350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3346714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3346874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3347247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3347441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3347690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3348000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3348404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3348805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3349037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3349270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3349356Z ok (5.760s)
2022-11-23T02:08:16.3349376Z 
2022-11-23T02:08:16.3349643Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3349758Z Ran 1 test in 5.760s
2022-11-23T02:08:16.3349778Z 
2022-11-23T02:08:16.3349873Z OK
2022-11-23T02:08:16.3349895Z 
2022-11-23T02:08:16.3350019Z Generating XML reports...
2022-11-23T02:08:16.3350469Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020531.xml
2022-11-23T02:08:16.3350840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3351017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3351396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3351618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3351638Z 
2022-11-23T02:08:16.3351756Z Running tests...
2022-11-23T02:08:16.3352021Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3352336Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3352595Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3352614Z 
2022-11-23T02:08:16.3352878Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3352995Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3353014Z 
2022-11-23T02:08:16.3353123Z OK (skipped=1)
2022-11-23T02:08:16.3353143Z 
2022-11-23T02:08:16.3353248Z Generating XML reports...
2022-11-23T02:08:16.3353700Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020539.xml
2022-11-23T02:08:16.3354073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3354257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3354644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3354840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3354859Z 
2022-11-23T02:08:16.3354967Z Running tests...
2022-11-23T02:08:16.3355228Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3355524Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3355779Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3355999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37695
2022-11-23T02:08:16.3356223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37696
2022-11-23T02:08:16.3356593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3356767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3357144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3357399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3357746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3357921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3358294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3358486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3358734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3358979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3359378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3359776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3360005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3360215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3360554Z STAGE:2022-11-23 02:05:47 37695:37695 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3360924Z STAGE:2022-11-23 02:05:47 37696:37696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3361276Z STAGE:2022-11-23 02:05:47 37696:37696 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3361603Z STAGE:2022-11-23 02:05:47 37695:37695 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3362176Z STAGE:2022-11-23 02:05:47 37696:37696 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:05:47 37695:37695 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3362200Z 
2022-11-23T02:08:16.3362528Z STAGE:2022-11-23 02:05:47 37696:37696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3362854Z STAGE:2022-11-23 02:05:47 37695:37695 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3363186Z STAGE:2022-11-23 02:05:47 37696:37696 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3363535Z STAGE:2022-11-23 02:05:47 37696:37696 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3363851Z STAGE:2022-11-23 02:05:47 37695:37695 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3364193Z STAGE:2022-11-23 02:05:47 37695:37695 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3364299Z ok (5.668s)
2022-11-23T02:08:16.3364319Z 
2022-11-23T02:08:16.3364584Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3364694Z Ran 1 test in 5.668s
2022-11-23T02:08:16.3364714Z 
2022-11-23T02:08:16.3364806Z OK
2022-11-23T02:08:16.3364825Z 
2022-11-23T02:08:16.3364948Z Generating XML reports...
2022-11-23T02:08:16.3365400Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020542.xml
2022-11-23T02:08:16.3365777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3365939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3366317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3366508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3366584Z 
2022-11-23T02:08:16.3366698Z Running tests...
2022-11-23T02:08:16.3366964Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3367280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3367547Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3367766Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37813
2022-11-23T02:08:16.3367971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37814
2022-11-23T02:08:16.3368342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3368516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3368892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3369086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3369453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3369629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3370002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3370177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3370475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3370731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3371135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3371711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3371954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3372186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3372526Z STAGE:2022-11-23 02:05:55 37814:37814 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3372855Z STAGE:2022-11-23 02:05:55 37813:37813 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3373178Z STAGE:2022-11-23 02:05:55 37814:37814 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3373517Z STAGE:2022-11-23 02:05:55 37813:37813 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3373863Z STAGE:2022-11-23 02:05:55 37814:37814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3374211Z STAGE:2022-11-23 02:05:55 37813:37813 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3374537Z STAGE:2022-11-23 02:05:55 37814:37814 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3374865Z STAGE:2022-11-23 02:05:55 37814:37814 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3375211Z STAGE:2022-11-23 02:05:55 37814:37814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3375540Z STAGE:2022-11-23 02:05:55 37813:37813 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3375871Z STAGE:2022-11-23 02:05:55 37813:37813 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3376197Z STAGE:2022-11-23 02:05:55 37813:37813 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3376301Z ok (5.756s)
2022-11-23T02:08:16.3376321Z 
2022-11-23T02:08:16.3376684Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3376798Z Ran 1 test in 5.757s
2022-11-23T02:08:16.3376817Z 
2022-11-23T02:08:16.3376911Z OK
2022-11-23T02:08:16.3376929Z 
2022-11-23T02:08:16.3377055Z Generating XML reports...
2022-11-23T02:08:16.3377507Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020550.xml
2022-11-23T02:08:16.3377882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3378063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3378425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3378617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3378636Z 
2022-11-23T02:08:16.3378742Z Running tests...
2022-11-23T02:08:16.3379006Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3379319Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3379581Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3379600Z 
2022-11-23T02:08:16.3379857Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3379967Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3379986Z 
2022-11-23T02:08:16.3380093Z OK (skipped=1)
2022-11-23T02:08:16.3380113Z 
2022-11-23T02:08:16.3380292Z Generating XML reports...
2022-11-23T02:08:16.3380763Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020558.xml
2022-11-23T02:08:16.3381137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3381312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3381695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3381888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3381907Z 
2022-11-23T02:08:16.3382016Z Running tests...
2022-11-23T02:08:16.3382280Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3382577Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3382830Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3382849Z 
2022-11-23T02:08:16.3383120Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3383231Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3383250Z 
2022-11-23T02:08:16.3383357Z OK (skipped=1)
2022-11-23T02:08:16.3383376Z 
2022-11-23T02:08:16.3383502Z Generating XML reports...
2022-11-23T02:08:16.3383953Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020601.xml
2022-11-23T02:08:16.3384325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3384500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3384862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3385056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3385075Z 
2022-11-23T02:08:16.3385181Z Running tests...
2022-11-23T02:08:16.3385441Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3385752Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3386016Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3386091Z 
2022-11-23T02:08:16.3386357Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3386472Z Ran 1 test in 0.003s
2022-11-23T02:08:16.3386491Z 
2022-11-23T02:08:16.3386599Z OK (skipped=1)
2022-11-23T02:08:16.3386618Z 
2022-11-23T02:08:16.3386723Z Generating XML reports...
2022-11-23T02:08:16.3387172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020603.xml
2022-11-23T02:08:16.3387545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3387719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3388098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3388287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3388310Z 
2022-11-23T02:08:16.3388421Z Running tests...
2022-11-23T02:08:16.3388683Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3388980Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3389246Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3389265Z 
2022-11-23T02:08:16.3389523Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3389682Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3389703Z 
2022-11-23T02:08:16.3389816Z OK (skipped=1)
2022-11-23T02:08:16.3389835Z 
2022-11-23T02:08:16.3389960Z Generating XML reports...
2022-11-23T02:08:16.3390411Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020605.xml
2022-11-23T02:08:16.3390782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3390961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3391324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3391514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3391533Z 
2022-11-23T02:08:16.3391640Z Running tests...
2022-11-23T02:08:16.3391902Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3392214Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3392464Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3392684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38063
2022-11-23T02:08:16.3392901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38064
2022-11-23T02:08:16.3393258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3393431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3393810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3394001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3394362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3394533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3394903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3395092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3395415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3395643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3396048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3396443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3396677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3396907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3397249Z STAGE:2022-11-23 02:06:13 38064:38064 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3397573Z STAGE:2022-11-23 02:06:13 38063:38063 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3397913Z STAGE:2022-11-23 02:06:13 38063:38063 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3398262Z STAGE:2022-11-23 02:06:13 38063:38063 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3398579Z STAGE:2022-11-23 02:06:13 38064:38064 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3398921Z STAGE:2022-11-23 02:06:13 38064:38064 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3399294Z STAGE:2022-11-23 02:06:14 38064:38064 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3399628Z STAGE:2022-11-23 02:06:14 38063:38063 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3399965Z STAGE:2022-11-23 02:06:14 38064:38064 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3400307Z STAGE:2022-11-23 02:06:14 38064:38064 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3400644Z STAGE:2022-11-23 02:06:14 38063:38063 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3400987Z STAGE:2022-11-23 02:06:14 38063:38063 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3401092Z ok (6.475s)
2022-11-23T02:08:16.3401112Z 
2022-11-23T02:08:16.3401357Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3401471Z Ran 1 test in 6.476s
2022-11-23T02:08:16.3401490Z 
2022-11-23T02:08:16.3401581Z OK
2022-11-23T02:08:16.3401603Z 
2022-11-23T02:08:16.3401727Z Generating XML reports...
2022-11-23T02:08:16.3402180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020608.xml
2022-11-23T02:08:16.3402552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3402731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3403111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3403286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3403322Z 
2022-11-23T02:08:16.3403414Z Running tests...
2022-11-23T02:08:16.3403726Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3404047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3404313Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3404532Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38186
2022-11-23T02:08:16.3404750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38187
2022-11-23T02:08:16.3405187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3405363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3405723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3405914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3406281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3406459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3406830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3407018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3407263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3407511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3407894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3408291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3408520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3408803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3409151Z STAGE:2022-11-23 02:06:22 38187:38187 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3409478Z STAGE:2022-11-23 02:06:22 38186:38186 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3409815Z STAGE:2022-11-23 02:06:22 38186:38186 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3410167Z STAGE:2022-11-23 02:06:22 38186:38186 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3410500Z STAGE:2022-11-23 02:06:22 38187:38187 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3410828Z STAGE:2022-11-23 02:06:22 38187:38187 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3411155Z STAGE:2022-11-23 02:06:23 38186:38186 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3411652Z STAGE:2022-11-23 02:06:23 38187:38187 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3412009Z STAGE:2022-11-23 02:06:23 38187:38187 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3412355Z STAGE:2022-11-23 02:06:23 38187:38187 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3412690Z STAGE:2022-11-23 02:06:23 38186:38186 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3413033Z STAGE:2022-11-23 02:06:23 38186:38186 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3413135Z ok (6.439s)
2022-11-23T02:08:16.3413155Z 
2022-11-23T02:08:16.3413418Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3413513Z Ran 1 test in 6.439s
2022-11-23T02:08:16.3413532Z 
2022-11-23T02:08:16.3413625Z OK
2022-11-23T02:08:16.3413644Z 
2022-11-23T02:08:16.3413766Z Generating XML reports...
2022-11-23T02:08:16.3414220Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020617.xml
2022-11-23T02:08:16.3414589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3414766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3415247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3415441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3415460Z 
2022-11-23T02:08:16.3415568Z Running tests...
2022-11-23T02:08:16.3415814Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3416129Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3416403Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3416423Z 
2022-11-23T02:08:16.3416682Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3416794Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3416814Z 
2022-11-23T02:08:16.3416922Z OK (skipped=1)
2022-11-23T02:08:16.3416941Z 
2022-11-23T02:08:16.3417065Z Generating XML reports...
2022-11-23T02:08:16.3417515Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020626.xml
2022-11-23T02:08:16.3417888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3418046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3418423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3418681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3418702Z 
2022-11-23T02:08:16.3418819Z Running tests...
2022-11-23T02:08:16.3419086Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3419402Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3419664Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-11-23T02:08:16.3419687Z 
2022-11-23T02:08:16.3419947Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3420041Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3420077Z 
2022-11-23T02:08:16.3420166Z OK (skipped=1)
2022-11-23T02:08:16.3420185Z 
2022-11-23T02:08:16.3420306Z Generating XML reports...
2022-11-23T02:08:16.3420754Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020628.xml
2022-11-23T02:08:16.3421125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3421300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3421681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3421875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3421899Z 
2022-11-23T02:08:16.3422009Z Running tests...
2022-11-23T02:08:16.3422253Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3422564Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3422946Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-11-23T02:08:16.3422965Z 
2022-11-23T02:08:16.3423225Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3423338Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3423357Z 
2022-11-23T02:08:16.3423462Z OK (skipped=1)
2022-11-23T02:08:16.3423481Z 
2022-11-23T02:08:16.3423605Z Generating XML reports...
2022-11-23T02:08:16.3424052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020631.xml
2022-11-23T02:08:16.3424424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3424642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3425027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3425218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3425237Z 
2022-11-23T02:08:16.3425346Z Running tests...
2022-11-23T02:08:16.3425610Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3425927Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3426191Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:08:16.3426210Z 
2022-11-23T02:08:16.3426469Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3426567Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3426603Z 
2022-11-23T02:08:16.3426693Z OK (skipped=1)
2022-11-23T02:08:16.3426713Z 
2022-11-23T02:08:16.3426836Z Generating XML reports...
2022-11-23T02:08:16.3427280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020633.xml
2022-11-23T02:08:16.3427650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3427825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3428249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3428448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3428467Z 
2022-11-23T02:08:16.3428576Z Running tests...
2022-11-23T02:08:16.3428824Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3429140Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3429424Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T02:08:16.3429444Z 
2022-11-23T02:08:16.3429709Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3429821Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3429840Z 
2022-11-23T02:08:16.3429945Z OK (skipped=1)
2022-11-23T02:08:16.3429964Z 
2022-11-23T02:08:16.3430086Z Generating XML reports...
2022-11-23T02:08:16.3430532Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020635.xml
2022-11-23T02:08:16.3430899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3431056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3431437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3431626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3431644Z 
2022-11-23T02:08:16.3431752Z Running tests...
2022-11-23T02:08:16.3432014Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3432324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3432640Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T02:08:16.3432660Z 
2022-11-23T02:08:16.3432922Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3433030Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3433050Z 
2022-11-23T02:08:16.3433141Z OK (skipped=1)
2022-11-23T02:08:16.3433159Z 
2022-11-23T02:08:16.3433280Z Generating XML reports...
2022-11-23T02:08:16.3433790Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020638.xml
2022-11-23T02:08:16.3434162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3434338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3434712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3434907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3434927Z 
2022-11-23T02:08:16.3435035Z Running tests...
2022-11-23T02:08:16.3435277Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3435593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3435896Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-11-23T02:08:16.3435920Z 
2022-11-23T02:08:16.3436173Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3436285Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3436305Z 
2022-11-23T02:08:16.3436411Z OK (skipped=1)
2022-11-23T02:08:16.3436430Z 
2022-11-23T02:08:16.3436553Z Generating XML reports...
2022-11-23T02:08:16.3436999Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020640.xml
2022-11-23T02:08:16.3437419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3437589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3437973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3438166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3438188Z 
2022-11-23T02:08:16.3438296Z Running tests...
2022-11-23T02:08:16.3438559Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3438873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3439166Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:08:16.3439185Z 
2022-11-23T02:08:16.3439450Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3439561Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3439580Z 
2022-11-23T02:08:16.3439668Z OK (skipped=1)
2022-11-23T02:08:16.3439704Z 
2022-11-23T02:08:16.3439809Z Generating XML reports...
2022-11-23T02:08:16.3440254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020643.xml
2022-11-23T02:08:16.3440624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3440797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3441174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3441365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3441384Z 
2022-11-23T02:08:16.3441491Z Running tests...
2022-11-23T02:08:16.3441756Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3442053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3442306Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3442526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38573
2022-11-23T02:08:16.3442807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38574
2022-11-23T02:08:16.3443182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3443358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3443736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3443926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3444275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3444449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3444824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3445010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3445259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3445503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3445902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3446299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3446578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3446796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3446903Z ok (5.160s)
2022-11-23T02:08:16.3446922Z 
2022-11-23T02:08:16.3447189Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3447306Z Ran 1 test in 5.160s
2022-11-23T02:08:16.3447326Z 
2022-11-23T02:08:16.3447423Z OK
2022-11-23T02:08:16.3447442Z 
2022-11-23T02:08:16.3447567Z Generating XML reports...
2022-11-23T02:08:16.3448017Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020645.xml
2022-11-23T02:08:16.3448385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3448543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3448924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3449113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3449132Z 
2022-11-23T02:08:16.3449239Z Running tests...
2022-11-23T02:08:16.3449499Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3449814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3450091Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3450309Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38691
2022-11-23T02:08:16.3450525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38692
2022-11-23T02:08:16.3450882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3451059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3451607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3451813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3452182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3452446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3452824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3453012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3453241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3453491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3453893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3454280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3454512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3454738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3455074Z STAGE:2022-11-23 02:06:58 38692:38692 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3455396Z STAGE:2022-11-23 02:06:58 38691:38691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3455732Z STAGE:2022-11-23 02:06:58 38692:38692 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3456150Z STAGE:2022-11-23 02:06:58 38691:38691 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3456516Z STAGE:2022-11-23 02:06:58 38692:38692 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3456861Z STAGE:2022-11-23 02:06:58 38691:38691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3456970Z ok (5.656s)
2022-11-23T02:08:16.3456989Z 
2022-11-23T02:08:16.3457253Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3457364Z Ran 1 test in 5.656s
2022-11-23T02:08:16.3457384Z 
2022-11-23T02:08:16.3457476Z OK
2022-11-23T02:08:16.3457495Z 
2022-11-23T02:08:16.3457619Z Generating XML reports...
2022-11-23T02:08:16.3458073Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020653.xml
2022-11-23T02:08:16.3458432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3458610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3458992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3459182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3459204Z 
2022-11-23T02:08:16.3459311Z Running tests...
2022-11-23T02:08:16.3459576Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3459888Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3460163Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3460365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38813
2022-11-23T02:08:16.3460585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38814
2022-11-23T02:08:16.3460961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3461137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3461516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3461786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3462153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3462328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3462700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3462874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3463125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3463368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3463767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3464163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3464396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3464625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3464960Z STAGE:2022-11-23 02:07:06 38813:38813 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3465279Z STAGE:2022-11-23 02:07:06 38814:38814 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:08:16.3465648Z STAGE:2022-11-23 02:07:06 38814:38814 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3466000Z STAGE:2022-11-23 02:07:06 38813:38813 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:08:16.3466347Z STAGE:2022-11-23 02:07:06 38814:38814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3466702Z STAGE:2022-11-23 02:07:06 38813:38813 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:08:16.3466809Z ok (5.756s)
2022-11-23T02:08:16.3466828Z 
2022-11-23T02:08:16.3467092Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3467204Z Ran 1 test in 5.756s
2022-11-23T02:08:16.3467223Z 
2022-11-23T02:08:16.3467316Z OK
2022-11-23T02:08:16.3467335Z 
2022-11-23T02:08:16.3467442Z Generating XML reports...
2022-11-23T02:08:16.3467898Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020701.xml
2022-11-23T02:08:16.3468268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3468441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3468818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3469016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3469035Z 
2022-11-23T02:08:16.3469143Z Running tests...
2022-11-23T02:08:16.3469406Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3469722Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3469993Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:08:16.3470031Z 
2022-11-23T02:08:16.3470279Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3470388Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3470407Z 
2022-11-23T02:08:16.3470515Z OK (skipped=1)
2022-11-23T02:08:16.3470534Z 
2022-11-23T02:08:16.3470656Z Generating XML reports...
2022-11-23T02:08:16.3471103Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020709.xml
2022-11-23T02:08:16.3471539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3471714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3472092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3472266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3472285Z 
2022-11-23T02:08:16.3472394Z Running tests...
2022-11-23T02:08:16.3472661Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3472974Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3473245Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:08:16.3473263Z 
2022-11-23T02:08:16.3473523Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3473634Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3473653Z 
2022-11-23T02:08:16.3473759Z OK (skipped=1)
2022-11-23T02:08:16.3473778Z 
2022-11-23T02:08:16.3473901Z Generating XML reports...
2022-11-23T02:08:16.3474335Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020712.xml
2022-11-23T02:08:16.3474705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3474933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3475327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3475575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3475595Z 
2022-11-23T02:08:16.3475713Z Running tests...
2022-11-23T02:08:16.3475983Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3476297Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3476580Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:08:16.3476616Z 
2022-11-23T02:08:16.3476862Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3476972Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3476992Z 
2022-11-23T02:08:16.3477102Z OK (skipped=1)
2022-11-23T02:08:16.3477121Z 
2022-11-23T02:08:16.3477243Z Generating XML reports...
2022-11-23T02:08:16.3477690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020714.xml
2022-11-23T02:08:16.3478057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3478236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3478614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3478786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3478822Z 
2022-11-23T02:08:16.3478914Z Running tests...
2022-11-23T02:08:16.3479175Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3479490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3479782Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-11-23T02:08:16.3479801Z 
2022-11-23T02:08:16.3480059Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3480172Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3480247Z 
2022-11-23T02:08:16.3480361Z OK (skipped=1)
2022-11-23T02:08:16.3480379Z 
2022-11-23T02:08:16.3480503Z Generating XML reports...
2022-11-23T02:08:16.3480937Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020716.xml
2022-11-23T02:08:16.3481307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3481484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3481868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3482059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3482078Z 
2022-11-23T02:08:16.3482186Z Running tests...
2022-11-23T02:08:16.3482448Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3482757Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3483045Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-11-23T02:08:16.3483064Z 
2022-11-23T02:08:16.3483307Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3483418Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3483437Z 
2022-11-23T02:08:16.3483543Z OK (skipped=1)
2022-11-23T02:08:16.3483562Z 
2022-11-23T02:08:16.3483685Z Generating XML reports...
2022-11-23T02:08:16.3484180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020719.xml
2022-11-23T02:08:16.3484562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3484740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3485120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3485314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3485334Z 
2022-11-23T02:08:16.3485425Z Running tests...
2022-11-23T02:08:16.3485687Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3485997Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3486287Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-11-23T02:08:16.3486310Z 
2022-11-23T02:08:16.3486569Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3486679Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3486699Z 
2022-11-23T02:08:16.3486803Z OK (skipped=1)
2022-11-23T02:08:16.3486822Z 
2022-11-23T02:08:16.3486944Z Generating XML reports...
2022-11-23T02:08:16.3487371Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020721.xml
2022-11-23T02:08:16.3487742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3487916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3488292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3488479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3488498Z 
2022-11-23T02:08:16.3488609Z Running tests...
2022-11-23T02:08:16.3488868Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3489174Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3489440Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3489703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39133
2022-11-23T02:08:16.3489925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39134
2022-11-23T02:08:16.3490297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3490469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3490849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3491043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3491403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3491757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3492126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3492320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3492570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3492816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3493219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3493692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3493937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3494165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3494268Z ok (6.065s)
2022-11-23T02:08:16.3494292Z 
2022-11-23T02:08:16.3494547Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3494660Z Ran 1 test in 6.065s
2022-11-23T02:08:16.3494679Z 
2022-11-23T02:08:16.3494773Z OK
2022-11-23T02:08:16.3494792Z 
2022-11-23T02:08:16.3494919Z Generating XML reports...
2022-11-23T02:08:16.3495366Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020723.xml
2022-11-23T02:08:16.3495736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3495917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3496296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3496486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3496505Z 
2022-11-23T02:08:16.3496595Z Running tests...
2022-11-23T02:08:16.3496861Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3497171Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3497448Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-11-23T02:08:16.3497467Z 
2022-11-23T02:08:16.3497727Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3497837Z Ran 1 test in 0.002s
2022-11-23T02:08:16.3497856Z 
2022-11-23T02:08:16.3497964Z OK (skipped=1)
2022-11-23T02:08:16.3497986Z 
2022-11-23T02:08:16.3498108Z Generating XML reports...
2022-11-23T02:08:16.3498538Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020732.xml
2022-11-23T02:08:16.3498906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3499162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3499548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3499740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3499759Z 
2022-11-23T02:08:16.3499867Z Running tests...
2022-11-23T02:08:16.3500125Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3500437Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3500691Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3500894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39285
2022-11-23T02:08:16.3501111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39286
2022-11-23T02:08:16.3501480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3501657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3502033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3502224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3502588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3502814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3503197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3503372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3503668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3503923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3504325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3504722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3504953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3505184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3505440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps884kxix
2022-11-23T02:08:16.3505694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps884kxix/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3505949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkprup419
2022-11-23T02:08:16.3506218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkprup419/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3506319Z ok (5.634s)
2022-11-23T02:08:16.3506340Z 
2022-11-23T02:08:16.3506606Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3506718Z Ran 1 test in 5.634s
2022-11-23T02:08:16.3506737Z 
2022-11-23T02:08:16.3506828Z OK
2022-11-23T02:08:16.3506847Z 
2022-11-23T02:08:16.3506970Z Generating XML reports...
2022-11-23T02:08:16.3507419Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020734.xml
2022-11-23T02:08:16.3507775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3507950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3508326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3508584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3508603Z 
2022-11-23T02:08:16.3508713Z Running tests...
2022-11-23T02:08:16.3508977Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3509289Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3509582Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3509787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39400
2022-11-23T02:08:16.3510008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39401
2022-11-23T02:08:16.3510378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3510557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3510932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3511121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3511482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3511653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3512078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3512256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3512505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3512750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3513158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3513553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3513781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3514010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3514272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppq7gjq6a
2022-11-23T02:08:16.3514542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppq7gjq6a/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3514780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fpvmtul
2022-11-23T02:08:16.3515044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fpvmtul/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3515828Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:08:16.3516605Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:08:16.3516779Z ok (6.065s)
2022-11-23T02:08:16.3516800Z 
2022-11-23T02:08:16.3517072Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3517185Z Ran 1 test in 6.066s
2022-11-23T02:08:16.3517204Z 
2022-11-23T02:08:16.3517298Z OK
2022-11-23T02:08:16.3517317Z 
2022-11-23T02:08:16.3517441Z Generating XML reports...
2022-11-23T02:08:16.3517897Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020743.xml
2022-11-23T02:08:16.3518273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3518449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3518810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3519005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3519024Z 
2022-11-23T02:08:16.3519130Z Running tests...
2022-11-23T02:08:16.3519389Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3519701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3519983Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3520253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39519
2022-11-23T02:08:16.3520483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39520
2022-11-23T02:08:16.3520841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3521017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3521398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3521591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3521954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3522127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3522500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3522685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3522932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3523161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3523566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3523959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3524187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3524413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3524654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.3524896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.3525293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3525682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3525981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:08:16.3526219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:08:16.3526614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.3527013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.3527275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjilrhhs
2022-11-23T02:08:16.3527553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjilrhhs/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3527809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3n3m4wwu
2022-11-23T02:08:16.3528081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3n3m4wwu/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3528186Z ok (10.720s)
2022-11-23T02:08:16.3528206Z 
2022-11-23T02:08:16.3528456Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3528570Z Ran 1 test in 10.720s
2022-11-23T02:08:16.3528589Z 
2022-11-23T02:08:16.3528683Z OK
2022-11-23T02:08:16.3528702Z 
2022-11-23T02:08:16.3528824Z Generating XML reports...
2022-11-23T02:08:16.3529321Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020751.xml
2022-11-23T02:08:16.3529707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3529887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3530273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3530469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3530489Z 
2022-11-23T02:08:16.3530581Z Running tests...
2022-11-23T02:08:16.3530845Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3531159Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-11-23T02:08:16.3531703Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:08:16.3531948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39641
2022-11-23T02:08:16.3532168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39642
2022-11-23T02:08:16.3532545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3532723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3533088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3533279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3533641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:08:16.3533813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:08:16.3534193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:08:16.3534381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:08:16.3534629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:08:16.3534875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:08:16.3535392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3535771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:08:16.3536007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:08:16.3536232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:08:16.3536555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:08:16.3536899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:08:16.3537307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3537703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:08:16.3537948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:08:16.3538186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:08:16.3538554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.3538945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:08:16.3539286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpplng9p0t
2022-11-23T02:08:16.3539577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpplng9p0t/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3539832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48j6c57y
2022-11-23T02:08:16.3540105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48j6c57y/_remote_module_non_scriptable.py
2022-11-23T02:08:16.3540211Z ok (10.542s)
2022-11-23T02:08:16.3540233Z 
2022-11-23T02:08:16.3540503Z ----------------------------------------------------------------------
2022-11-23T02:08:16.3540599Z Ran 1 test in 10.542s
2022-11-23T02:08:16.3540636Z 
2022-11-23T02:08:16.3540826Z OK
2022-11-23T02:08:16.3540845Z 
2022-11-23T02:08:16.3540974Z Generating XML reports...
2022-11-23T02:08:16.3541427Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020804.xml
2022-11-23T02:08:16.3541450Z 
2022-11-23T02:08:16.3541918Z ##[endgroup]
2022-11-23T02:08:16.3542390Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_07am423u)
2022-11-23T02:08:16.3542411Z 
2022-11-23T02:08:16.3542540Z Shard 2: gloo should be run in 1
2022-11-23T02:08:16.3542667Z Shard 2: ucc should be run in 3
2022-11-23T02:08:16.3542936Z Running distributed/rpc/test_tensorpipe_agent ... [2022-11-23 02:08:16.141047]
2022-11-23T02:08:16.3543453Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:08:16.141366]
2022-11-23T02:08:18.6946904Z 
2022-11-23T02:08:18.6947705Z Expand the folded group to see the log file of distributed/rpc/test_tensorpipe_agent
2022-11-23T02:08:18.6949388Z ##[group]PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_1_2gykv6)
2022-11-23T02:08:18.6950122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe12zc99t
2022-11-23T02:08:18.6951318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe12zc99t/_remote_module_non_scriptable.py
2022-11-23T02:08:18.6951712Z 
2022-11-23T02:08:18.6952029Z ##[endgroup]
2022-11-23T02:08:18.6953736Z FINISHED PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_1_2gykv6)
2022-11-23T02:08:18.6954129Z 
2022-11-23T02:08:18.6954456Z Running distributed/pipeline/sync/test_transparency ... [2022-11-23 02:08:18.694735]
2022-11-23T02:08:18.6955092Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-11-23 02:08:18.694990]
2022-11-23T02:08:21.7183475Z 
2022-11-23T02:08:21.7184026Z Expand the folded group to see the log file of distributed/pipeline/sync/test_transparency
2022-11-23T02:08:21.7185123Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_xpq91f9a)
2022-11-23T02:08:21.7185758Z ============================= test session starts ==============================
2022-11-23T02:08:21.7186430Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:08:21.7187018Z cachedir: .pytest_cache
2022-11-23T02:08:21.7187685Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:08:21.7188194Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:08:21.7188543Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:08:21.7189228Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:08:21.7189712Z collecting ... collected 1 item
2022-11-23T02:08:21.7190348Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears
2022-11-23T02:08:21.7190694Z 
2022-11-23T02:08:21.7190960Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2022-11-23T02:08:21.7191256Z 
2022-11-23T02:08:21.7191455Z ============================== 1 passed in 0.14s ===============================
2022-11-23T02:08:21.7191754Z 
2022-11-23T02:08:21.7192139Z ##[endgroup]
2022-11-23T02:08:21.7205291Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_xpq91f9a)
2022-11-23T02:08:21.7205772Z 
2022-11-23T02:08:21.7206112Z Running distributed/pipeline/sync/test_pipe ... [2022-11-23 02:08:21.718500]
2022-11-23T02:08:21.7206737Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-11-23 02:08:21.718764]
2022-11-23T02:08:29.1377920Z 
2022-11-23T02:08:29.1378550Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipe
2022-11-23T02:08:29.1380312Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_st_2rz4v)
2022-11-23T02:08:29.1381423Z ============================= test session starts ==============================
2022-11-23T02:08:29.1382607Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:08:29.1382991Z cachedir: .pytest_cache
2022-11-23T02:08:29.1383567Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:08:29.1383994Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:08:29.1384328Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:08:29.1385074Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:08:29.1385993Z collecting ... collected 56 items
2022-11-23T02:08:29.1392182Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper
2022-11-23T02:08:29.1397762Z 
2022-11-23T02:08:29.1397972Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED     [  1%]
2022-11-23T02:08:29.1398418Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  3%]
2022-11-23T02:08:29.1398845Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
2022-11-23T02:08:29.1399342Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  7%]
2022-11-23T02:08:29.1399754Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [  8%]
2022-11-23T02:08:29.1400194Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%]
2022-11-23T02:08:29.1400639Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 12%]
2022-11-23T02:08:29.1401047Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 14%]
2022-11-23T02:08:29.1401501Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%]
2022-11-23T02:08:29.1401973Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%]
2022-11-23T02:08:29.1402431Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 19%]
2022-11-23T02:08:29.1402866Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%]
2022-11-23T02:08:29.1403308Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 23%]
2022-11-23T02:08:29.1403725Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 25%]
2022-11-23T02:08:29.1404217Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%]
2022-11-23T02:08:29.1404668Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 28%]
2022-11-23T02:08:29.1405103Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 30%]
2022-11-23T02:08:29.1405589Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%]
2022-11-23T02:08:29.1406014Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 33%]
2022-11-23T02:08:29.1406435Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 35%]
2022-11-23T02:08:29.1406858Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 37%]
2022-11-23T02:08:29.1407267Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 39%]
2022-11-23T02:08:29.1407713Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%]
2022-11-23T02:08:29.1408166Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%]
2022-11-23T02:08:29.1408626Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%]
2022-11-23T02:08:29.1409068Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%]
2022-11-23T02:08:29.1409515Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%]
2022-11-23T02:08:29.1409977Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%]
2022-11-23T02:08:29.1410437Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%]
2022-11-23T02:08:29.1410877Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%]
2022-11-23T02:08:29.1411337Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%]
2022-11-23T02:08:29.1412109Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED      [ 57%]
2022-11-23T02:08:29.1412529Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED     [ 58%]
2022-11-23T02:08:29.1412966Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%]
2022-11-23T02:08:29.1413420Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%]
2022-11-23T02:08:29.1413884Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%]
2022-11-23T02:08:29.1414345Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%]
2022-11-23T02:08:29.1414827Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%]
2022-11-23T02:08:29.1415311Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
2022-11-23T02:08:29.1415853Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 71%]
2022-11-23T02:08:29.1416280Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 73%]
2022-11-23T02:08:29.1416714Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions PASSED    [ 75%]
2022-11-23T02:08:29.1417144Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 76%]
2022-11-23T02:08:29.1417551Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 78%]
2022-11-23T02:08:29.1417972Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 80%]
2022-11-23T02:08:29.1418428Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%]
2022-11-23T02:08:29.1418888Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%]
2022-11-23T02:08:29.1419372Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device PASSED [ 85%]
2022-11-23T02:08:29.1419839Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules PASSED [ 87%]
2022-11-23T02:08:29.1420336Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%]
2022-11-23T02:08:29.1420800Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [ 91%]
2022-11-23T02:08:29.1421246Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] PASSED [ 92%]
2022-11-23T02:08:29.1421704Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] PASSED [ 94%]
2022-11-23T02:08:29.1422219Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] PASSED [ 96%]
2022-11-23T02:08:29.1422694Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device PASSED  [ 98%]
2022-11-23T02:08:29.1423131Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper PASSED  [100%]
2022-11-23T02:08:29.1423382Z 
2022-11-23T02:08:29.1423541Z ============================== 56 passed in 4.89s ==============================
2022-11-23T02:08:29.1423746Z 
2022-11-23T02:08:29.1424059Z ##[endgroup]
2022-11-23T02:08:29.1424715Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_st_2rz4v)
2022-11-23T02:08:29.1425092Z 
2022-11-23T02:08:29.1425375Z Running distributed/pipeline/sync/test_inplace ... [2022-11-23 02:08:29.137841]
2022-11-23T02:08:29.1425972Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-11-23 02:08:29.138213]
2022-11-23T02:08:31.7147038Z 
2022-11-23T02:08:31.7147833Z Expand the folded group to see the log file of distributed/pipeline/sync/test_inplace
2022-11-23T02:08:31.7149051Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_wy4kpe9h)
2022-11-23T02:08:31.7149598Z ============================= test session starts ==============================
2022-11-23T02:08:31.7150513Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:08:31.7151024Z cachedir: .pytest_cache
2022-11-23T02:08:31.7151596Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:08:31.7152039Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:08:31.7152370Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:08:31.7153247Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:08:31.7153725Z collecting ... collected 3 items
2022-11-23T02:08:31.7154370Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad
2022-11-23T02:08:31.7154858Z 
2022-11-23T02:08:31.7155091Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2022-11-23T02:08:31.7155848Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2022-11-23T02:08:31.7156301Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2022-11-23T02:08:31.7156560Z 
2022-11-23T02:08:31.7156729Z ========================= 1 passed, 2 xfailed in 0.18s =========================
2022-11-23T02:08:31.7156934Z 
2022-11-23T02:08:31.7157256Z ##[endgroup]
2022-11-23T02:08:31.7157910Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_wy4kpe9h)
2022-11-23T02:08:31.7158297Z 
2022-11-23T02:08:31.7158575Z Running distributed/pipeline/sync/test_copy ... [2022-11-23 02:08:31.714788]
2022-11-23T02:08:31.7159188Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-11-23 02:08:31.715098]
2022-11-23T02:08:36.5143401Z 
2022-11-23T02:08:36.5144151Z Expand the folded group to see the log file of distributed/pipeline/sync/test_copy
2022-11-23T02:08:36.5145249Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_51l1qe_y)
2022-11-23T02:08:36.5145773Z ============================= test session starts ==============================
2022-11-23T02:08:36.5146399Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:08:36.5146788Z cachedir: .pytest_cache
2022-11-23T02:08:36.5147618Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:08:36.5148072Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:08:36.5148409Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:08:36.5148997Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:08:36.5149397Z collecting ... collected 5 items
2022-11-23T02:08:36.5150171Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors
2022-11-23T02:08:36.5150788Z 
2022-11-23T02:08:36.5151009Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
2022-11-23T02:08:36.5151462Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda PASSED   [ 40%]
2022-11-23T02:08:36.5151901Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu PASSED   [ 60%]
2022-11-23T02:08:36.5152315Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda PASSED  [ 80%]
2022-11-23T02:08:36.5152755Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]
2022-11-23T02:08:36.5153012Z 
2022-11-23T02:08:36.5153174Z ============================== 5 passed in 2.35s ===============================
2022-11-23T02:08:36.5153369Z 
2022-11-23T02:08:36.5155397Z ##[endgroup]
2022-11-23T02:08:36.5156078Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_51l1qe_y)
2022-11-23T02:08:36.5156460Z 
2022-11-23T02:08:36.5156749Z Running distributed/pipeline/sync/test_balance ... [2022-11-23 02:08:36.514336]
2022-11-23T02:08:36.5157388Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-11-23 02:08:36.514637]
2022-11-23T02:08:45.9489696Z 
2022-11-23T02:08:45.9490751Z Expand the folded group to see the log file of distributed/pipeline/sync/test_balance
2022-11-23T02:08:45.9492716Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_o64wirwf)
2022-11-23T02:08:45.9493767Z ============================= test session starts ==============================
2022-11-23T02:08:45.9494390Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:08:45.9494750Z cachedir: .pytest_cache
2022-11-23T02:08:45.9495350Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:08:45.9495777Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:08:45.9496119Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:08:45.9497076Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:08:45.9497557Z collecting ... collected 18 items
2022-11-23T02:08:45.9499799Z Running 18 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad
2022-11-23T02:08:45.9501888Z 
2022-11-23T02:08:45.9502116Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  5%]
2022-11-23T02:08:45.9502565Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%]
2022-11-23T02:08:45.9503068Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%]
2022-11-23T02:08:45.9503573Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%]
2022-11-23T02:08:45.9504045Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%]
2022-11-23T02:08:45.9504484Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%]
2022-11-23T02:08:45.9505018Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%]
2022-11-23T02:08:45.9505494Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%]
2022-11-23T02:08:45.9505931Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%]
2022-11-23T02:08:45.9506395Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%]
2022-11-23T02:08:45.9506861Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%]
2022-11-23T02:08:45.9507320Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%]
2022-11-23T02:08:45.9507774Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%]
2022-11-23T02:08:45.9508252Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%]
2022-11-23T02:08:45.9508711Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 83%]
2022-11-23T02:08:45.9509234Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%]
2022-11-23T02:08:45.9509668Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%]
2022-11-23T02:08:45.9510117Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2022-11-23T02:08:45.9510361Z 
2022-11-23T02:08:45.9510533Z ======================== 16 passed, 2 skipped in 6.88s =========================
2022-11-23T02:08:45.9510740Z 
2022-11-23T02:08:45.9511043Z ##[endgroup]
2022-11-23T02:08:45.9511708Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_o64wirwf)
2022-11-23T02:08:45.9512089Z 
2022-11-23T02:08:45.9512389Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-11-23 02:08:45.949043]
2022-11-23T02:08:45.9513031Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-11-23 02:08:45.949386]
2022-11-23T02:08:48.4013313Z 
2022-11-23T02:08:48.4014461Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_stash_pop
2022-11-23T02:08:48.4015804Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_xqxz171_)
2022-11-23T02:08:48.4016352Z ============================= test session starts ==============================
2022-11-23T02:08:48.4016952Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:08:48.4017771Z cachedir: .pytest_cache
2022-11-23T02:08:48.4018376Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:08:48.4018822Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:08:48.4019156Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:08:48.4019737Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:08:48.4020440Z collecting ... collected 7 items
2022-11-23T02:08:48.4021450Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none
2022-11-23T02:08:48.4022264Z 
2022-11-23T02:08:48.4022479Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
2022-11-23T02:08:48.4022914Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
2022-11-23T02:08:48.4023368Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
2022-11-23T02:08:48.4023822Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
2022-11-23T02:08:48.4024283Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
2022-11-23T02:08:48.4024739Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
2022-11-23T02:08:48.4025170Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]
2022-11-23T02:08:48.4025420Z 
2022-11-23T02:08:48.4025581Z ============================== 7 passed in 0.06s ===============================
2022-11-23T02:08:48.4025776Z 
2022-11-23T02:08:48.4026089Z ##[endgroup]
2022-11-23T02:08:48.4026753Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_xqxz171_)
2022-11-23T02:08:48.4027161Z 
2022-11-23T02:08:48.4027484Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-11-23 02:08:48.401459]
2022-11-23T02:08:48.4028295Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-11-23 02:08:48.401778]
2022-11-23T02:08:50.8415305Z 
2022-11-23T02:08:50.8415806Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-11-23T02:08:50.8417099Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_rg1hqr0d)
2022-11-23T02:08:50.8417712Z ============================= test session starts ==============================
2022-11-23T02:08:50.8418359Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:08:50.8418700Z cachedir: .pytest_cache
2022-11-23T02:08:50.8419286Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:08:50.8419750Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:08:50.8420062Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:08:50.8420646Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:08:50.8421054Z collecting ... collected 6 items
2022-11-23T02:08:50.8422322Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace
2022-11-23T02:08:50.8423189Z 
2022-11-23T02:08:50.8423437Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2022-11-23T02:08:50.8423911Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2022-11-23T02:08:50.8424414Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2022-11-23T02:08:50.8424909Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2022-11-23T02:08:50.8425424Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2022-11-23T02:08:50.8425909Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2022-11-23T02:08:50.8426174Z 
2022-11-23T02:08:50.8426335Z ============================== 6 passed in 0.04s ===============================
2022-11-23T02:08:50.8426532Z 
2022-11-23T02:08:50.8426844Z ##[endgroup]
2022-11-23T02:08:50.8427553Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_rg1hqr0d)
2022-11-23T02:08:50.8427991Z 
2022-11-23T02:08:50.8428310Z Running distributed/optim/test_apply_optimizer_in_backward ... [2022-11-23 02:08:50.841669]
2022-11-23T02:08:50.8429048Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:08:50.841938]
2022-11-23T02:08:52.9835131Z 
2022-11-23T02:08:52.9835662Z Expand the folded group to see the log file of distributed/optim/test_apply_optimizer_in_backward
2022-11-23T02:08:52.9836656Z ##[group]PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_mle_w073)
2022-11-23T02:08:52.9837068Z 
2022-11-23T02:08:52.9837367Z ##[endgroup]
2022-11-23T02:08:52.9838143Z FINISHED PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_mle_w073)
2022-11-23T02:08:52.9838819Z 
2022-11-23T02:08:52.9839400Z Running distributed/elastic/events/lib_test ... [2022-11-23 02:08:52.983638]
2022-11-23T02:08:52.9842356Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-11-23 02:08:52.983954]
2022-11-23T02:08:55.4959870Z 
2022-11-23T02:08:55.4960355Z Expand the folded group to see the log file of distributed/elastic/events/lib_test
2022-11-23T02:08:55.4961657Z ##[group]PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_glgnk5me)
2022-11-23T02:08:55.4962458Z ============================= test session starts ==============================
2022-11-23T02:08:55.4963477Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:08:55.4964286Z cachedir: .pytest_cache
2022-11-23T02:08:55.4964992Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:08:55.4965484Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:08:55.4966060Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:08:55.4966468Z collecting ... collected 8 items
2022-11-23T02:08:55.4967958Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str
2022-11-23T02:08:55.4969067Z 
2022-11-23T02:08:55.4969298Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%]
2022-11-23T02:08:55.4969762Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%]
2022-11-23T02:08:55.4970219Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%]
2022-11-23T02:08:55.4970726Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%]
2022-11-23T02:08:55.4971291Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%]
2022-11-23T02:08:55.4972598Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%]
2022-11-23T02:08:55.4973113Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%]
2022-11-23T02:08:55.4973603Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%]
2022-11-23T02:08:55.4973871Z 
2022-11-23T02:08:55.4974037Z ============================== 8 passed in 1.78s ===============================
2022-11-23T02:08:55.4974234Z 
2022-11-23T02:08:55.4974554Z ##[endgroup]
2022-11-23T02:08:55.4975180Z FINISHED PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_glgnk5me)
2022-11-23T02:08:55.4975550Z 
2022-11-23T02:08:55.4975816Z Running distributed/_shard/test_sharder ... [2022-11-23 02:08:55.496061]
2022-11-23T02:08:55.4976494Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:08:55.496328]
2022-11-23T02:08:57.6330755Z 
2022-11-23T02:08:57.6331901Z Expand the folded group to see the log file of distributed/_shard/test_sharder
2022-11-23T02:08:57.6333646Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_cvog0c7v)
2022-11-23T02:08:57.6334506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3scpcc9x
2022-11-23T02:08:57.6335058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3scpcc9x/_remote_module_non_scriptable.py
2022-11-23T02:08:57.6335451Z 
2022-11-23T02:08:57.6336076Z ##[endgroup]
2022-11-23T02:08:57.6337037Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_cvog0c7v)
2022-11-23T02:08:57.6337392Z 
2022-11-23T02:08:57.6337687Z Running distributed/nn/jit/test_instantiator ... [2022-11-23 02:08:57.633219]
2022-11-23T02:08:57.6339330Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:08:57.633584]
2022-11-23T02:09:01.5757579Z 
2022-11-23T02:09:01.5758474Z Expand the folded group to see the log file of distributed/nn/jit/test_instantiator
2022-11-23T02:09:01.5760334Z ##[group]PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_9kaohkaw)
2022-11-23T02:09:01.5761143Z 
2022-11-23T02:09:01.5761303Z Running tests...
2022-11-23T02:09:01.5762049Z ----------------------------------------------------------------------
2022-11-23T02:09:01.5762656Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2022-11-23T02:09:01.5763148Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (1.635s)
2022-11-23T02:09:01.5763601Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2022-11-23T02:09:01.5764091Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.015s)
2022-11-23T02:09:01.5764355Z 
2022-11-23T02:09:01.5764629Z ----------------------------------------------------------------------
2022-11-23T02:09:01.5764968Z Ran 3 tests in 1.652s
2022-11-23T02:09:01.5765116Z 
2022-11-23T02:09:01.5765214Z OK
2022-11-23T02:09:01.5765353Z 
2022-11-23T02:09:01.5765479Z Generating XML reports...
2022-11-23T02:09:01.5766102Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221123020859.xml
2022-11-23T02:09:01.5766462Z 
2022-11-23T02:09:01.5766760Z ##[endgroup]
2022-11-23T02:09:01.5767374Z FINISHED PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_9kaohkaw)
2022-11-23T02:09:01.5767751Z 
2022-11-23T02:09:01.5768033Z Running distributed/checkpoint/test_planner ... [2022-11-23 02:09:01.575769]
2022-11-23T02:09:01.5768714Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:01.576104]
2022-11-23T02:09:05.5535629Z 
2022-11-23T02:09:05.5536190Z Expand the folded group to see the log file of distributed/checkpoint/test_planner
2022-11-23T02:09:05.5537184Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_xnf8nd2z)
2022-11-23T02:09:05.5537571Z 
2022-11-23T02:09:05.5537687Z Running tests...
2022-11-23T02:09:05.5538231Z ----------------------------------------------------------------------
2022-11-23T02:09:05.5538794Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_planner
2022-11-23T02:09:05.5539249Z   test_global_plan (__main__.TestSavePlan) ... ok (1.649s)
2022-11-23T02:09:05.5539632Z   test_load_with_resharding (__main__.TestSavePlan) ... ok (0.004s)
2022-11-23T02:09:05.5540325Z   test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s)
2022-11-23T02:09:05.5540701Z   test_local_load_plan (__main__.TestSavePlan) ... ok (0.004s)
2022-11-23T02:09:05.5541064Z   test_local_plan (__main__.TestSavePlan) ... ok (0.004s)
2022-11-23T02:09:05.5541273Z 
2022-11-23T02:09:05.5541578Z ----------------------------------------------------------------------
2022-11-23T02:09:05.5541894Z Ran 5 tests in 1.664s
2022-11-23T02:09:05.5542063Z 
2022-11-23T02:09:05.5542162Z OK
2022-11-23T02:09:05.5542300Z 
2022-11-23T02:09:05.5542428Z Generating XML reports...
2022-11-23T02:09:05.5543098Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20221123020903.xml
2022-11-23T02:09:05.5543616Z 
2022-11-23T02:09:05.5543949Z ##[endgroup]
2022-11-23T02:09:05.5544578Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_xnf8nd2z)
2022-11-23T02:09:05.5544957Z 
2022-11-23T02:09:05.5545218Z Running distributed/fsdp/test_utils ... [2022-11-23 02:09:05.553614]
2022-11-23T02:09:05.5545869Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:05.553907]
2022-11-23T02:09:09.5028345Z 
2022-11-23T02:09:09.5029085Z Expand the folded group to see the log file of distributed/fsdp/test_utils
2022-11-23T02:09:09.5030358Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_bho5mtwv)
2022-11-23T02:09:09.5030998Z 
2022-11-23T02:09:09.5031232Z Running tests...
2022-11-23T02:09:09.5031856Z ----------------------------------------------------------------------
2022-11-23T02:09:09.5032391Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-11-23T02:09:09.5032853Z   test_module_wrap_policy (__main__.TestGetSubmoduleToStates)
2022-11-23T02:09:09.5033294Z Tests the module wrap policy on a nested model with buffers and a ... ok (1.654s)
2022-11-23T02:09:09.5033999Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... ok (0.004s)
2022-11-23T02:09:09.5034491Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s)
2022-11-23T02:09:09.5034977Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... ok (0.004s)
2022-11-23T02:09:09.5035342Z   test_packed_sequence (__main__.TestUtils)
2022-11-23T02:09:09.5035703Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.003s)
2022-11-23T02:09:09.5036103Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2022-11-23T02:09:09.5036319Z 
2022-11-23T02:09:09.5036590Z ----------------------------------------------------------------------
2022-11-23T02:09:09.5036927Z Ran 6 tests in 1.669s
2022-11-23T02:09:09.5037074Z 
2022-11-23T02:09:09.5037172Z OK
2022-11-23T02:09:09.5037307Z 
2022-11-23T02:09:09.5037437Z Generating XML reports...
2022-11-23T02:09:09.5038064Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20221123020907.xml
2022-11-23T02:09:09.5038790Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221123020907.xml
2022-11-23T02:09:09.5039129Z 
2022-11-23T02:09:09.5039446Z ##[endgroup]
2022-11-23T02:09:09.5040016Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_bho5mtwv)
2022-11-23T02:09:09.5040356Z 
2022-11-23T02:09:09.5040634Z Running distributed/test_multi_threaded_pg ... [2022-11-23 02:09:09.502848]
2022-11-23T02:09:09.5041308Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:09.503135]
2022-11-23T02:09:13.5115463Z 
2022-11-23T02:09:13.5116170Z Expand the folded group to see the log file of distributed/test_multi_threaded_pg
2022-11-23T02:09:13.5117859Z ##[group]PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_multi_threaded_pg_vvhuz2sc)
2022-11-23T02:09:13.5118520Z 
2022-11-23T02:09:13.5118697Z Running tests...
2022-11-23T02:09:13.5119484Z ----------------------------------------------------------------------
2022-11-23T02:09:13.5120352Z Test results will be stored in test-reports/python-unittest/distributed.test_multi_threaded_pg
2022-11-23T02:09:13.5121281Z   test_allgather (__main__.TestCollectivesWithBaseClass) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:09:13.5122192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:13.5122760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:13.5123258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:09:13.5123742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:09:13.5124750Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5125866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5127238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5128508Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5129271Z ok (1.669s)
2022-11-23T02:09:13.5129808Z   test_broadcast (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:13.5130394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:13.5130880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:09:13.5131371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:09:13.5132340Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5133033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5133764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5134431Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5134833Z ok (0.020s)
2022-11-23T02:09:13.5135317Z   test_broadcast_object_list (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:13.5135901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:13.5136399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:09:13.5136892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:09:13.5137554Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5138226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5138907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5139590Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5140131Z 3 -> 4
2022-11-23T02:09:13.5140356Z 0 -> 4
2022-11-23T02:09:13.5140593Z 1 -> 4
2022-11-23T02:09:13.5140832Z 2 -> 4
2022-11-23T02:09:13.5141034Z ok (0.017s)
2022-11-23T02:09:13.5141511Z   test_reduce_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:13.5142104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:13.5142585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:09:13.5143081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:09:13.5143738Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5144426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5145090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5145773Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5146172Z ok (0.015s)
2022-11-23T02:09:13.5146712Z   test_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:13.5147291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:13.5147785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:09:13.5148273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:09:13.5148918Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5149602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5150287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5150972Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5151341Z ok (0.014s)
2022-11-23T02:09:13.5151824Z   test_broadcast_object_list (__main__.TestCollectivesWithWrapper) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:13.5152418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:13.5152916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:09:13.5153389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:09:13.5154043Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5154728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5155414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5156077Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:09:13.5156472Z ok (0.016s)
2022-11-23T02:09:13.5156622Z 
2022-11-23T02:09:13.5156895Z ----------------------------------------------------------------------
2022-11-23T02:09:13.5157279Z Ran 6 tests in 1.752s
2022-11-23T02:09:13.5157447Z 
2022-11-23T02:09:13.5157543Z OK
2022-11-23T02:09:13.5157679Z 
2022-11-23T02:09:13.5157807Z Generating XML reports...
2022-11-23T02:09:13.5158459Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20221123020911.xml
2022-11-23T02:09:13.5159281Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20221123020911.xml
2022-11-23T02:09:13.5159660Z 
2022-11-23T02:09:13.5159990Z ##[endgroup]
2022-11-23T02:09:13.5160588Z FINISHED PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_multi_threaded_pg_vvhuz2sc)
2022-11-23T02:09:13.5160939Z 
2022-11-23T02:09:13.5161195Z Running distributed/elastic/utils/util_test ... [2022-11-23 02:09:13.511604]
2022-11-23T02:09:13.5161885Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:13.511919]
2022-11-23T02:09:17.5255136Z 
2022-11-23T02:09:17.5255816Z Expand the folded group to see the log file of distributed/elastic/utils/util_test
2022-11-23T02:09:17.5256907Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_qewq0tij)
2022-11-23T02:09:17.5257288Z 
2022-11-23T02:09:17.5257385Z Running tests...
2022-11-23T02:09:17.5257916Z ----------------------------------------------------------------------
2022-11-23T02:09:17.5258744Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test
2022-11-23T02:09:17.5259364Z   test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (1.633s)
2022-11-23T02:09:17.5259786Z   test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s)
2022-11-23T02:09:17.5260152Z   test_synchronize (__main__.StoreUtilTest) ... ok (0.003s)
2022-11-23T02:09:17.5260671Z   test_get_logger (__main__.UtilTest) ... ok (0.110s)
2022-11-23T02:09:17.5261137Z   test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s)
2022-11-23T02:09:17.5261625Z   test_get_logger_different (__main__.UtilTest) ... ok (0.001s)
2022-11-23T02:09:17.5261990Z   test_get_logger_none (__main__.UtilTest) ... ok (0.001s)
2022-11-23T02:09:17.5262206Z 
2022-11-23T02:09:17.5262674Z ----------------------------------------------------------------------
2022-11-23T02:09:17.5263025Z Ran 7 tests in 1.752s
2022-11-23T02:09:17.5263191Z 
2022-11-23T02:09:17.5263288Z OK
2022-11-23T02:09:17.5263425Z 
2022-11-23T02:09:17.5263559Z Generating XML reports...
2022-11-23T02:09:17.5264167Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221123020915.xml
2022-11-23T02:09:17.5264924Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221123020915.xml
2022-11-23T02:09:17.5265278Z 
2022-11-23T02:09:17.5265601Z ##[endgroup]
2022-11-23T02:09:17.5266197Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_qewq0tij)
2022-11-23T02:09:17.5266567Z 
2022-11-23T02:09:17.5266838Z Running distributed/rpc/test_share_memory ... [2022-11-23 02:09:17.525629]
2022-11-23T02:09:17.5267557Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:17.525910]
2022-11-23T02:09:26.3270127Z 
2022-11-23T02:09:26.3270688Z Expand the folded group to see the log file of distributed/rpc/test_share_memory
2022-11-23T02:09:26.3272606Z ##[group]PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_wb2hf4l_)
2022-11-23T02:09:26.3273125Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:09:26.3273534Z <unittest.suite.TestSuite tests=[<__main__.TestRPCPickler testMethod=test_case>]>
2022-11-23T02:09:26.3274176Z test_case (__main__.TestRPCPickler)
2022-11-23T02:09:26.3274828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:26.3275289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:26.3275879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:26.3276339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:26.3276578Z 
2022-11-23T02:09:26.3276699Z Running tests...
2022-11-23T02:09:26.3277111Z ----------------------------------------------------------------------
2022-11-23T02:09:26.3277664Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory
2022-11-23T02:09:26.3278373Z   test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:26.3278879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:26.3279457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:26.3279934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:26.3280907Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:09:26.3281754Z   warnings.warn(message, UserWarning)
2022-11-23T02:09:26.3282039Z ok (4.418s)
2022-11-23T02:09:26.3282195Z 
2022-11-23T02:09:26.3282479Z ----------------------------------------------------------------------
2022-11-23T02:09:26.3282791Z Ran 1 test in 4.419s
2022-11-23T02:09:26.3282973Z 
2022-11-23T02:09:26.3283078Z OK
2022-11-23T02:09:26.3283218Z 
2022-11-23T02:09:26.3283346Z Generating XML reports...
2022-11-23T02:09:26.3283917Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20221123020921.xml
2022-11-23T02:09:26.3284262Z 
2022-11-23T02:09:26.3284589Z ##[endgroup]
2022-11-23T02:09:26.3285188Z FINISHED PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_wb2hf4l_)
2022-11-23T02:09:26.3285543Z 
2022-11-23T02:09:26.3285836Z Running distributed/elastic/utils/distributed_test ... [2022-11-23 02:09:26.327120]
2022-11-23T02:09:26.3286536Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:26.327434]
2022-11-23T02:09:33.4631015Z 
2022-11-23T02:09:33.4631889Z Expand the folded group to see the log file of distributed/elastic/utils/distributed_test
2022-11-23T02:09:33.4633669Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_1nh58bb4)
2022-11-23T02:09:33.4634480Z 
2022-11-23T02:09:33.4634660Z Running tests...
2022-11-23T02:09:33.4635548Z ----------------------------------------------------------------------
2022-11-23T02:09:33.4636811Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test
2022-11-23T02:09:33.4637765Z   test_create_store_multi (__main__.DistributedUtilTest) ... ok (1.698s)
2022-11-23T02:09:33.4638638Z   test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s)
2022-11-23T02:09:33.4639793Z   test_create_store_single_server (__main__.DistributedUtilTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/66207 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-11-23T02:09:33.4640860Z   test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.024s)
2022-11-23T02:09:33.4641422Z   test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (9de1665d20b3, 0).
2022-11-23T02:09:33.4641865Z ok (0.001s)
2022-11-23T02:09:33.4642501Z   test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:32985 (errno: 98 - Address already in use).
2022-11-23T02:09:33.4643191Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:32985 (errno: 98 - Address already in use).
2022-11-23T02:09:33.4643665Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2022-11-23T02:09:33.4643984Z ok (0.004s)
2022-11-23T02:09:33.4644450Z   test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (9de1665d20b3, 59457).
2022-11-23T02:09:33.4644882Z ok (0.003s)
2022-11-23T02:09:33.4645031Z 
2022-11-23T02:09:33.4645304Z ----------------------------------------------------------------------
2022-11-23T02:09:33.4645621Z Ran 7 tests in 4.734s
2022-11-23T02:09:33.4645787Z 
2022-11-23T02:09:33.4645896Z OK (skipped=1)
2022-11-23T02:09:33.4646052Z 
2022-11-23T02:09:33.4646178Z Generating XML reports...
2022-11-23T02:09:33.4646907Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20221123020928.xml
2022-11-23T02:09:33.4647315Z 
2022-11-23T02:09:33.4647641Z ##[endgroup]
2022-11-23T02:09:33.4648300Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_1nh58bb4)
2022-11-23T02:09:33.4648697Z 
2022-11-23T02:09:33.4648958Z Running distributed/fsdp/test_fsdp_fx ... [2022-11-23 02:09:33.463083]
2022-11-23T02:09:33.4649617Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:33.463426]
2022-11-23T02:09:40.7287630Z 
2022-11-23T02:09:40.7288321Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx
2022-11-23T02:09:40.7289248Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_0ji0xcxq)
2022-11-23T02:09:40.7289648Z 
2022-11-23T02:09:40.7289762Z Running tests...
2022-11-23T02:09:40.7290278Z ----------------------------------------------------------------------
2022-11-23T02:09:40.7290828Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx
2022-11-23T02:09:40.7291269Z   test_symbolic_tracing_outputs (__main__.TestSymbolicTracing)
2022-11-23T02:09:40.7292080Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:09:40.7292607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43221
2022-11-23T02:09:40.7293041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43222
2022-11-23T02:09:40.7293688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:40.7294146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:40.7294731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:40.7295191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:40.7295774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:40.7296522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:40.7297103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:40.7297551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:40.7298016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:40.7298521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:40.7299175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:09:40.7299870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:09:40.7300398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:09:40.7300875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:09:40.7301220Z dist init r=0, world=2
2022-11-23T02:09:40.7301475Z dist init r=1, world=2
2022-11-23T02:09:40.7301719Z ok (4.947s)
2022-11-23T02:09:40.7301870Z 
2022-11-23T02:09:40.7302144Z ----------------------------------------------------------------------
2022-11-23T02:09:40.7302474Z Ran 1 test in 4.947s
2022-11-23T02:09:40.7302637Z 
2022-11-23T02:09:40.7302733Z OK
2022-11-23T02:09:40.7302870Z 
2022-11-23T02:09:40.7302996Z Generating XML reports...
2022-11-23T02:09:40.7303676Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123020935.xml
2022-11-23T02:09:40.7304043Z 
2022-11-23T02:09:40.7304365Z ##[endgroup]
2022-11-23T02:09:40.7304950Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_0ji0xcxq)
2022-11-23T02:09:40.7305275Z 
2022-11-23T02:09:40.7305579Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-11-23 02:09:40.728740]
2022-11-23T02:09:40.7306299Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:40.729112]
2022-11-23T02:09:48.4713529Z 
2022-11-23T02:09:48.4714317Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T02:09:48.4715421Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_p2ns4eyq)
2022-11-23T02:09:48.4715827Z 
2022-11-23T02:09:48.4715946Z Running tests...
2022-11-23T02:09:48.4716493Z ----------------------------------------------------------------------
2022-11-23T02:09:48.4717067Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2022-11-23T02:09:48.4717585Z   test_multi_forward (__main__.TestMultiForward) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:09:48.4718071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43335
2022-11-23T02:09:48.4718525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43336
2022-11-23T02:09:48.4719134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:48.4719593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:48.4720180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:48.4720640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:48.4721221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:48.4721672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:48.4722534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:48.4722988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:48.4723448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:48.4723951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:48.4724621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:09:48.4725301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:09:48.4725828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:09:48.4726306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:09:48.4726773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:09:48.4727257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:09:48.4728631Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:09:48.4729434Z   warnings.warn(
2022-11-23T02:09:48.4730595Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:09:48.4731376Z   warnings.warn(
2022-11-23T02:09:48.4731922Z dist init r=1, world=2
2022-11-23T02:09:48.4732184Z dist init r=0, world=2
2022-11-23T02:09:48.4732428Z ok (5.411s)
2022-11-23T02:09:48.4732577Z 
2022-11-23T02:09:48.4732838Z ----------------------------------------------------------------------
2022-11-23T02:09:48.4733173Z Ran 1 test in 5.411s
2022-11-23T02:09:48.4733342Z 
2022-11-23T02:09:48.4733443Z OK
2022-11-23T02:09:48.4733581Z 
2022-11-23T02:09:48.4733703Z Generating XML reports...
2022-11-23T02:09:48.4734303Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123020942.xml
2022-11-23T02:09:48.4734670Z 
2022-11-23T02:09:48.4734997Z ##[endgroup]
2022-11-23T02:09:48.4735646Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_p2ns4eyq)
2022-11-23T02:09:48.4736013Z 
2022-11-23T02:09:48.4736324Z Running distributed/_tensor/parallel/test_2d_parallel ... [2022-11-23 02:09:48.471505]
2022-11-23T02:09:48.4737049Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_2d_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:48.471838]
2022-11-23T02:09:57.2902300Z 
2022-11-23T02:09:57.2903119Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_2d_parallel
2022-11-23T02:09:57.2905035Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_ee_9nm3b)
2022-11-23T02:09:57.2905700Z 
2022-11-23T02:09:57.2905806Z Running tests...
2022-11-23T02:09:57.2906367Z ----------------------------------------------------------------------
2022-11-23T02:09:57.2907299Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel
2022-11-23T02:09:57.2908255Z   test_2d_fsdp_integration_correctness (__main__.Test2dParallelIntegration) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:09:57.2908858Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43453
2022-11-23T02:09:57.2909324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43454
2022-11-23T02:09:57.2909966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:57.2910422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:57.2910995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:57.2911468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:57.2912058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:57.2912494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:57.2913070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:57.2913539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:57.2913985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:09:57.2914590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:57.2915100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:09:57.2915588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:57.2916256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:09:57.2916922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:09:57.2917369Z skip: Need at least 4 CUDA devices (3.993s)
2022-11-23T02:09:57.2917883Z   test_2d_fsdp_integration_functionality (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43523
2022-11-23T02:09:57.2918435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43524
2022-11-23T02:09:57.2919044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:57.2919497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:57.2920074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:57.2920535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:57.2921112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:09:57.2921563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:09:57.2922133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:09:57.2922611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:09:57.2923059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:09:57.2923547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:09:57.2924034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:09:57.2924499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:09:57.2925249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:09:57.2925942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:09:57.2926369Z skip: Need at least 4 CUDA devices (2.408s)
2022-11-23T02:09:57.2926565Z 
2022-11-23T02:09:57.2926836Z ----------------------------------------------------------------------
2022-11-23T02:09:57.2927177Z Ran 2 tests in 6.401s
2022-11-23T02:09:57.2927340Z 
2022-11-23T02:09:57.2927451Z OK (skipped=2)
2022-11-23T02:09:57.2927589Z 
2022-11-23T02:09:57.2927713Z Generating XML reports...
2022-11-23T02:09:57.2928366Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20221123020950.xml
2022-11-23T02:09:57.2928756Z 
2022-11-23T02:09:57.2929088Z ##[endgroup]
2022-11-23T02:09:57.2929726Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_ee_9nm3b)
2022-11-23T02:09:57.2930114Z 
2022-11-23T02:09:57.2930446Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-11-23 02:09:57.290346]
2022-11-23T02:09:57.2931214Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:09:57.290634]
2022-11-23T02:10:06.0604540Z 
2022-11-23T02:10:06.0605128Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T02:10:06.0606298Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_u7leg64y)
2022-11-23T02:10:06.0606739Z 
2022-11-23T02:10:06.0606889Z Running tests...
2022-11-23T02:10:06.0607412Z ----------------------------------------------------------------------
2022-11-23T02:10:06.0608022Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard
2022-11-23T02:10:06.0608559Z   test_sharded_tensor_reshard (__main__.TestReshard) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:10:06.0609090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43628
2022-11-23T02:10:06.0609535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43629
2022-11-23T02:10:06.0609987Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43630
2022-11-23T02:10:06.0610426Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43631
2022-11-23T02:10:06.0611058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:06.0611823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:06.0612437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:06.0612915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:06.0613477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:06.0613930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:06.0614509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:06.0614978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:06.0615542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:06.0616172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:06.0616756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:06.0617222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:06.0617785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:06.0618233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:06.0618807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:06.0619260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:06.0619706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:10:06.0620187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:06.0620660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:06.0621115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:10:06.0621507Z skip: Need at least 4 CUDA devices (4.048s)
2022-11-23T02:10:06.0621989Z   test_sharded_tensor_reshard_errors (__main__.TestReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43764
2022-11-23T02:10:06.0622497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43765
2022-11-23T02:10:06.0623040Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43766
2022-11-23T02:10:06.0623500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43767
2022-11-23T02:10:06.0624123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:06.0624562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:06.0625144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:06.0625616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:06.0626200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:06.0626656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:06.0627241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:06.0627712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:06.0628274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:06.0628725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:06.0629307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:06.0629774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:06.0630336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:06.0630785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:06.0631362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:06.0631828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:06.0632252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:10:06.0632727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:10:06.0633202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:06.0633735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:06.0634139Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:10:06.0634336Z 
2022-11-23T02:10:06.0634616Z ----------------------------------------------------------------------
2022-11-23T02:10:06.0634954Z Ran 2 tests in 6.459s
2022-11-23T02:10:06.0635100Z 
2022-11-23T02:10:06.0635211Z OK (skipped=2)
2022-11-23T02:10:06.0635370Z 
2022-11-23T02:10:06.0635496Z Generating XML reports...
2022-11-23T02:10:06.0636129Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221123020959.xml
2022-11-23T02:10:06.0636494Z 
2022-11-23T02:10:06.0636809Z ##[endgroup]
2022-11-23T02:10:06.0637521Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_u7leg64y)
2022-11-23T02:10:06.0637950Z 
2022-11-23T02:10:06.0638236Z Running distributed/_composable/test_replicate ... [2022-11-23 02:10:06.060484]
2022-11-23T02:10:06.0638943Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:10:06.060766]
2022-11-23T02:10:14.9621796Z 
2022-11-23T02:10:14.9622823Z Expand the folded group to see the log file of distributed/_composable/test_replicate
2022-11-23T02:10:14.9624972Z ##[group]PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_9a7gh90b)
2022-11-23T02:10:14.9625719Z 
2022-11-23T02:10:14.9625845Z Running tests...
2022-11-23T02:10:14.9626381Z ----------------------------------------------------------------------
2022-11-23T02:10:14.9626935Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_replicate
2022-11-23T02:10:14.9627458Z   test_replicate_multi_module (__main__.ReplicateTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:10:14.9627932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43935
2022-11-23T02:10:14.9628366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43936
2022-11-23T02:10:14.9628814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43937
2022-11-23T02:10:14.9629250Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43938
2022-11-23T02:10:14.9629880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:14.9630320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:14.9630900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:14.9631376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:14.9631944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:14.9632395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:14.9632971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:14.9633438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:14.9634000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:14.9634446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:14.9635013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:14.9635478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:14.9636193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:14.9636638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:14.9637207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:14.9637655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:14.9638093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:10:14.9638573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:14.9639045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:14.9639496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:10:14.9639979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:10:14.9640480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:10:14.9640951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:10:14.9641438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:10:14.9642095Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:10:14.9642844Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:10:14.9643528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:10:14.9644209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:10:14.9644739Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:10:14.9645219Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:10:14.9645675Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:10:14.9646149Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:10:14.9646492Z ok (4.083s)
2022-11-23T02:10:14.9646964Z   test_replicate_single_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44091
2022-11-23T02:10:14.9647474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44092
2022-11-23T02:10:14.9647921Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44093
2022-11-23T02:10:14.9648362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44094
2022-11-23T02:10:14.9648979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:14.9649413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:14.9649989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:14.9650459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:14.9651023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:14.9651757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:14.9652350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:14.9652816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:14.9653481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:14.9653936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:14.9654504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:14.9654968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:14.9655529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:14.9655973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:14.9656543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:14.9656986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:14.9657434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:10:14.9657907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:10:14.9658375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:14.9658824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:14.9659306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:10:14.9659875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:10:14.9660362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:10:14.9660858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:10:14.9661519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:10:14.9662210Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:10:14.9662877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:10:14.9663557Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:10:14.9664088Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:10:14.9664565Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:10:14.9665018Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:10:14.9665493Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:10:14.9665837Z ok (2.510s)
2022-11-23T02:10:14.9665987Z 
2022-11-23T02:10:14.9666242Z ----------------------------------------------------------------------
2022-11-23T02:10:14.9666578Z Ran 2 tests in 6.593s
2022-11-23T02:10:14.9666744Z 
2022-11-23T02:10:14.9666839Z OK
2022-11-23T02:10:14.9666977Z 
2022-11-23T02:10:14.9667102Z Generating XML reports...
2022-11-23T02:10:14.9667674Z Generated XML report: test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20221123021007.xml
2022-11-23T02:10:14.9668021Z 
2022-11-23T02:10:14.9668340Z ##[endgroup]
2022-11-23T02:10:14.9668965Z FINISHED PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_9a7gh90b)
2022-11-23T02:10:14.9669334Z 
2022-11-23T02:10:14.9669614Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-11-23 02:10:14.962237]
2022-11-23T02:10:14.9670336Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:10:14.962549]
2022-11-23T02:10:25.5054693Z 
2022-11-23T02:10:25.5055652Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint
2022-11-23T02:10:25.5057007Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_tb36l2xd)
2022-11-23T02:10:25.5057806Z 
2022-11-23T02:10:25.5058084Z Running tests...
2022-11-23T02:10:25.5059047Z ----------------------------------------------------------------------
2022-11-23T02:10:25.5060315Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2022-11-23T02:10:25.5061128Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:10:25.5061709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44282
2022-11-23T02:10:25.5062147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44283
2022-11-23T02:10:25.5062775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:25.5063232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:25.5063795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:25.5064515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:25.5065130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:25.5065579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:25.5066136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:25.5066615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:25.5067072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:10:25.5067580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:10:25.5068227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:10:25.5068953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:10:25.5069482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:25.5069958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:25.5070300Z dist init r=0, world=2
2022-11-23T02:10:25.5070563Z dist init r=1, world=2
2022-11-23T02:10:25.5070807Z ok (4.910s)
2022-11-23T02:10:25.5071312Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44365
2022-11-23T02:10:25.5071925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44366
2022-11-23T02:10:25.5072542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:25.5073001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:25.5073562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:25.5074035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:25.5074618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:25.5075188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:25.5075752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:25.5076219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:25.5076682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:10:25.5077163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:10:25.5077833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:10:25.5078523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:10:25.5079050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:25.5079515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:25.5079866Z dist init r=1, world=2
2022-11-23T02:10:25.5080119Z dist init r=0, world=2
2022-11-23T02:10:25.5080345Z ok (3.310s)
2022-11-23T02:10:25.5080495Z 
2022-11-23T02:10:25.5080770Z ----------------------------------------------------------------------
2022-11-23T02:10:25.5081106Z Ran 2 tests in 8.220s
2022-11-23T02:10:25.5081272Z 
2022-11-23T02:10:25.5081347Z OK
2022-11-23T02:10:25.5081484Z 
2022-11-23T02:10:25.5081615Z Generating XML reports...
2022-11-23T02:10:25.5082329Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123021016.xml
2022-11-23T02:10:25.5082738Z 
2022-11-23T02:10:25.5083065Z ##[endgroup]
2022-11-23T02:10:25.5083702Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_tb36l2xd)
2022-11-23T02:10:25.5084099Z 
2022-11-23T02:10:25.5084395Z Running distributed/elastic/timer/local_timer_example ... [2022-11-23 02:10:25.505509]
2022-11-23T02:10:25.5085114Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:10:25.505797]
2022-11-23T02:10:38.9464229Z 
2022-11-23T02:10:38.9464972Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_example
2022-11-23T02:10:38.9466017Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_e7iwzt66)
2022-11-23T02:10:38.9466409Z 
2022-11-23T02:10:38.9466556Z Running tests...
2022-11-23T02:10:38.9467092Z ----------------------------------------------------------------------
2022-11-23T02:10:38.9467697Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2022-11-23T02:10:38.9468369Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-11-23 02:10:28,924 driver: init
2022-11-23T02:10:38.9468915Z [INFO] 2022-11-23 02:10:28,993 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-11-23T02:10:38.9469385Z [INFO] 2022-11-23 02:10:28,993 api: Starting watchdog thread...
2022-11-23T02:10:38.9469945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9470389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9470976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9471448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9472035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9472767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9473356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9473835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9474419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9474859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9475439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9475931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9476516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9476955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9477528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9477995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9478560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9479013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9479701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9480194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9480760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9481209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9481776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9482229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9482786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9483256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9483847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9484299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9484876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9485326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9485903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9486354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9486848Z [INFO] 2022-11-23 02:10:31,012 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9487336Z [INFO] 2022-11-23 02:10:31,041 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9487800Z [INFO] 2022-11-23 02:10:31,047 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9488285Z [INFO] 2022-11-23 02:10:31,055 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9488769Z [INFO] 2022-11-23 02:10:31,069 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9489248Z [INFO] 2022-11-23 02:10:31,074 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9489708Z [INFO] 2022-11-23 02:10:31,078 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9490179Z [INFO] 2022-11-23 02:10:31,080 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9490843Z [INFO] 2022-11-23 02:10:32,169 api: Reaping worker_id=[44488]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9491346Z [INFO] 2022-11-23 02:10:32,170 api: Successfully reaped worker=[44488]
2022-11-23T02:10:38.9492271Z [INFO] 2022-11-23 02:10:32,190 api: Reaping worker_id=[44484]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9492784Z [INFO] 2022-11-23 02:10:32,191 api: Successfully reaped worker=[44484]
2022-11-23T02:10:38.9493349Z [INFO] 2022-11-23 02:10:32,191 api: Reaping worker_id=[44490]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9493835Z [INFO] 2022-11-23 02:10:32,191 api: Successfully reaped worker=[44490]
2022-11-23T02:10:38.9494391Z [INFO] 2022-11-23 02:10:32,201 api: Reaping worker_id=[44486]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9494904Z [INFO] 2022-11-23 02:10:32,202 api: Successfully reaped worker=[44486]
2022-11-23T02:10:38.9495334Z [INFO] 2022-11-23 02:10:32,212 api: Stopping LocalTimerServer
2022-11-23T02:10:38.9495744Z [INFO] 2022-11-23 02:10:32,212 api: Stopping watchdog thread...
2022-11-23T02:10:38.9496035Z ok (4.842s)
2022-11-23T02:10:38.9496600Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-11-23 02:10:32,224 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-11-23T02:10:38.9497117Z [INFO] 2022-11-23 02:10:32,225 api: Starting watchdog thread...
2022-11-23T02:10:38.9497774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9498249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9498829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9499285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9499871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9500317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9500871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9501340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9501917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9502361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9502971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9503395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9503953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9504400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9504970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9505423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9506006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9506472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9507036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9507502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9508081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9508625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9509189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9509657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9510228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9510706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9511282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9511749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9512321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9512751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9513319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9513781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9514270Z [INFO] 2022-11-23 02:10:34,275 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9514742Z [INFO] 2022-11-23 02:10:34,284 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9515280Z [INFO] 2022-11-23 02:10:34,337 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9515779Z [INFO] 2022-11-23 02:10:34,338 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9516241Z [INFO] 2022-11-23 02:10:34,340 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9516710Z [INFO] 2022-11-23 02:10:34,347 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9517188Z [INFO] 2022-11-23 02:10:34,359 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9517658Z [INFO] 2022-11-23 02:10:34,374 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9518215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9518662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9519237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9519690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9520269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9520713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9521284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9521740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9522313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9522754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9523305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9523773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9524344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9524783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9525333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9525904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9526478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9526916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9527466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9527928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9528499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9528920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9529485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9529950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9530520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9530941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9531748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9532220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9532863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:38.9533321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:38.9533895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:38.9534356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:38.9534829Z [INFO] 2022-11-23 02:10:37,351 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9535318Z [INFO] 2022-11-23 02:10:37,377 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9535801Z [INFO] 2022-11-23 02:10:37,390 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9536260Z [INFO] 2022-11-23 02:10:37,397 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9536740Z [INFO] 2022-11-23 02:10:37,415 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9537218Z [INFO] 2022-11-23 02:10:37,419 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9537694Z [INFO] 2022-11-23 02:10:37,437 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9538150Z [INFO] 2022-11-23 02:10:37,448 api: Timer client configured to: LocalTimerClient
2022-11-23T02:10:38.9538719Z [INFO] 2022-11-23 02:10:38,459 api: Reaping worker_id=[45033]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9539238Z [INFO] 2022-11-23 02:10:38,459 api: Successfully reaped worker=[45033]
2022-11-23T02:10:38.9539799Z [INFO] 2022-11-23 02:10:38,490 api: Reaping worker_id=[45035]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9540290Z [INFO] 2022-11-23 02:10:38,490 api: Successfully reaped worker=[45035]
2022-11-23T02:10:38.9540852Z [INFO] 2022-11-23 02:10:38,510 api: Reaping worker_id=[45029]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9541404Z [INFO] 2022-11-23 02:10:38,510 local_timer: Process with pid=45029 does not exist. Skipping
2022-11-23T02:10:38.9541863Z [INFO] 2022-11-23 02:10:38,510 api: Successfully reaped worker=[45029]
2022-11-23T02:10:38.9542422Z [INFO] 2022-11-23 02:10:38,510 api: Reaping worker_id=[45032]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9542929Z [INFO] 2022-11-23 02:10:38,510 api: Successfully reaped worker=[45032]
2022-11-23T02:10:38.9543582Z [INFO] 2022-11-23 02:10:38,531 api: Reaping worker_id=[45034]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9544108Z [INFO] 2022-11-23 02:10:38,531 local_timer: Process with pid=45034 does not exist. Skipping
2022-11-23T02:10:38.9544582Z [INFO] 2022-11-23 02:10:38,531 api: Successfully reaped worker=[45034]
2022-11-23T02:10:38.9545018Z [INFO] 2022-11-23 02:10:38,533 api: Stopping LocalTimerServer
2022-11-23T02:10:38.9545428Z [INFO] 2022-11-23 02:10:38,533 api: Stopping watchdog thread...
2022-11-23T02:10:38.9545985Z [INFO] 2022-11-23 02:10:38,541 api: Reaping worker_id=[45036]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:10:38.9546535Z [INFO] 2022-11-23 02:10:38,541 local_timer: Process with pid=45036 does not exist. Skipping
2022-11-23T02:10:38.9547004Z [INFO] 2022-11-23 02:10:38,541 api: Successfully reaped worker=[45036]
2022-11-23T02:10:38.9547290Z ok (6.319s)
2022-11-23T02:10:38.9547443Z 
2022-11-23T02:10:38.9547712Z ----------------------------------------------------------------------
2022-11-23T02:10:38.9548045Z Ran 2 tests in 11.161s
2022-11-23T02:10:38.9548210Z 
2022-11-23T02:10:38.9548288Z OK
2022-11-23T02:10:38.9548421Z 
2022-11-23T02:10:38.9548546Z Generating XML reports...
2022-11-23T02:10:38.9549179Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221123021027.xml
2022-11-23T02:10:38.9549560Z 
2022-11-23T02:10:38.9549878Z ##[endgroup]
2022-11-23T02:10:38.9550600Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_e7iwzt66)
2022-11-23T02:10:38.9551010Z 
2022-11-23T02:10:38.9551336Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-11-23 02:10:38.946484]
2022-11-23T02:10:38.9552091Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:10:38.946775]
2022-11-23T02:10:52.5996846Z 
2022-11-23T02:10:52.5997739Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-11-23T02:10:52.5998914Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_qlyuk3cq)
2022-11-23T02:10:52.5999350Z 
2022-11-23T02:10:52.5999474Z Running tests...
2022-11-23T02:10:52.5999992Z ----------------------------------------------------------------------
2022-11-23T02:10:52.6000613Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2022-11-23T02:10:52.6001197Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:10:52.6001709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45336
2022-11-23T02:10:52.6002164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45337
2022-11-23T02:10:52.6002608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45338
2022-11-23T02:10:52.6003093Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45339
2022-11-23T02:10:52.6003706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6004167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6004741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6005193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6005756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6006518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6007114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6007632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6008196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6008653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6009235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6009704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6010270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6010725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6011305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6011836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6012588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:52.6013070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:10:52.6013646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:52.6014115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:10:52.6014507Z skip: Need at least 4 CUDA devices (4.036s)
2022-11-23T02:10:52.6015016Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45472
2022-11-23T02:10:52.6015575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45473
2022-11-23T02:10:52.6016005Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45474
2022-11-23T02:10:52.6016446Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45475
2022-11-23T02:10:52.6017075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6017510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6018089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6018559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6019146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6019571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6020151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6020616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6021201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6021631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6022206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6022670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6023228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6023676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6024346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6024815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6025239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:10:52.6025716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:52.6026186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:10:52.6026643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:52.6027041Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:10:52.6027550Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45608
2022-11-23T02:10:52.6028101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45609
2022-11-23T02:10:52.6028537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45610
2022-11-23T02:10:52.6028975Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45611
2022-11-23T02:10:52.6029590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6030025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6030663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6031144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6031731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6032160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6032740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6033204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6033778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6034204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6034778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6035247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6035807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6036251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6036821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6037287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6037707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:10:52.6038183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:52.6038653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:52.6039106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:10:52.6039501Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:10:52.6040021Z   test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45744
2022-11-23T02:10:52.6040587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45745
2022-11-23T02:10:52.6041089Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45746
2022-11-23T02:10:52.6041525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45747
2022-11-23T02:10:52.6042142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6042593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6043151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6043627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6044207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6044634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6045204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6045680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6046254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6046676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6047245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6047769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6048336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:10:52.6048778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:10:52.6049345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:10:52.6049811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:10:52.6050230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:10:52.6050708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:10:52.6051174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:10:52.6051836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:10:52.6052218Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:10:52.6052418Z 
2022-11-23T02:10:52.6052695Z ----------------------------------------------------------------------
2022-11-23T02:10:52.6053031Z Ran 4 tests in 11.367s
2022-11-23T02:10:52.6053197Z 
2022-11-23T02:10:52.6053288Z OK (skipped=4)
2022-11-23T02:10:52.6053444Z 
2022-11-23T02:10:52.6053573Z Generating XML reports...
2022-11-23T02:10:52.6054293Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221123021040.xml
2022-11-23T02:10:52.6054720Z 
2022-11-23T02:10:52.6055030Z ##[endgroup]
2022-11-23T02:10:52.6055734Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_qlyuk3cq)
2022-11-23T02:10:52.6056153Z 
2022-11-23T02:10:52.6056428Z Running distributed/fsdp/test_fsdp_apply ... [2022-11-23 02:10:52.599734]
2022-11-23T02:10:52.6057115Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:10:52.599999]
2022-11-23T02:11:07.0820872Z 
2022-11-23T02:11:07.0821912Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply
2022-11-23T02:11:07.0823539Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_hyiv6a2g)
2022-11-23T02:11:07.0824554Z 
2022-11-23T02:11:07.0824762Z Running tests...
2022-11-23T02:11:07.0825636Z ----------------------------------------------------------------------
2022-11-23T02:11:07.0826680Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2022-11-23T02:11:07.0827506Z   test_apply_in_summon_raises_error (__main__.TestApply)
2022-11-23T02:11:07.0828337Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:11:07.0829214Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45915
2022-11-23T02:11:07.0829806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45916
2022-11-23T02:11:07.0830467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:07.0830935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:07.0831507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:07.0831988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:07.0832683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:07.0833464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:07.0834207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:07.0834710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:07.0835170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:11:07.0835674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:11:07.0836333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:07.0837027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:07.0837556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:07.0838012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:07.0839288Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:11:07.0840085Z   warnings.warn(
2022-11-23T02:11:07.0841233Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:11:07.0842015Z   warnings.warn(
2022-11-23T02:11:07.0842287Z   File "<string>", line 1, in <module>
2022-11-23T02:11:07.0842646Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:11:07.0843025Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:11:07.0843398Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:11:07.0843838Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:11:07.0844232Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:11:07.0844571Z     self.run()
2022-11-23T02:11:07.0844909Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:11:07.0845260Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:11:07.0845784Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:11:07.0846176Z     self.run_test(test_name, pipe)
2022-11-23T02:11:07.0846691Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:11:07.0847092Z     getattr(self, test_name)()
2022-11-23T02:11:07.0847610Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:11:07.0847961Z     fn()
2022-11-23T02:11:07.0848454Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:11:07.0848845Z     return func(*args, **kwargs)
2022-11-23T02:11:07.0849268Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_apply.py", line 98, in test_apply_in_summon_raises_error
2022-11-23T02:11:07.0849695Z     transformer.apply(self._init_linear_weights)
2022-11-23T02:11:07.0850261Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 492, in apply
2022-11-23T02:11:07.0850756Z     self._assert_state(TrainingState.IDLE)
2022-11-23T02:11:07.0851327Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1084, in _assert_state
2022-11-23T02:11:07.0852293Z     traceback.print_stack()
2022-11-23T02:11:07.0852570Z dist init r=1, world=2
2022-11-23T02:11:07.0852823Z dist init r=0, world=2
2022-11-23T02:11:07.0853131Z Asserting FSDP instance is: FullyShardedDataParallel(
2022-11-23T02:11:07.0853535Z   (_fsdp_wrapped_module): TransformerWithSharedParams(
2022-11-23T02:11:07.0853879Z     (embed_tokens): Embedding(23, 16)
2022-11-23T02:11:07.0854156Z     (transformer): Transformer(
2022-11-23T02:11:07.0854451Z       (encoder): TransformerEncoder(
2022-11-23T02:11:07.0854736Z         (layers): ModuleList(
2022-11-23T02:11:07.0855016Z           (0): FullyShardedDataParallel(
2022-11-23T02:11:07.0855459Z             (_fsdp_wrapped_module): TransformerEncoderLayer(
2022-11-23T02:11:07.0855792Z               (self_attn): MultiheadAttention(
2022-11-23T02:11:07.0856214Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:11:07.0856576Z               )
2022-11-23T02:11:07.0856868Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-11-23T02:11:07.0857230Z               (dropout): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0857588Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-11-23T02:11:07.0858052Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0858512Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0858870Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0859203Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0859458Z             )
2022-11-23T02:11:07.0859684Z           )
2022-11-23T02:11:07.0859958Z           (1): FullyShardedDataParallel(
2022-11-23T02:11:07.0860305Z             (_fsdp_wrapped_module): TransformerEncoderLayer(
2022-11-23T02:11:07.0860659Z               (self_attn): MultiheadAttention(
2022-11-23T02:11:07.0861079Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:11:07.0861420Z               )
2022-11-23T02:11:07.0861725Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-11-23T02:11:07.0862207Z               (dropout): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0862546Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-11-23T02:11:07.0863016Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0863474Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0863822Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0864140Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0864414Z             )
2022-11-23T02:11:07.0864643Z           )
2022-11-23T02:11:07.0864846Z         )
2022-11-23T02:11:07.0865220Z         (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0865518Z       )
2022-11-23T02:11:07.0865759Z       (decoder): TransformerDecoder(
2022-11-23T02:11:07.0866041Z         (layers): ModuleList(
2022-11-23T02:11:07.0866344Z           (0): FullyShardedDataParallel(
2022-11-23T02:11:07.0866702Z             (_fsdp_wrapped_module): TransformerDecoderLayer(
2022-11-23T02:11:07.0867033Z               (self_attn): MultiheadAttention(
2022-11-23T02:11:07.0867450Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:11:07.0867803Z               )
2022-11-23T02:11:07.0868088Z               (multihead_attn): MultiheadAttention(
2022-11-23T02:11:07.0868492Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:11:07.0868842Z               )
2022-11-23T02:11:07.0869227Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-11-23T02:11:07.0869583Z               (dropout): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0869942Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-11-23T02:11:07.0870401Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0870845Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0871305Z               (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0871652Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0871981Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0872293Z               (dropout3): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0872567Z             )
2022-11-23T02:11:07.0872788Z           )
2022-11-23T02:11:07.0873042Z           (1): FullyShardedDataParallel(
2022-11-23T02:11:07.0873403Z             (_fsdp_wrapped_module): TransformerDecoderLayer(
2022-11-23T02:11:07.0873752Z               (self_attn): MultiheadAttention(
2022-11-23T02:11:07.0874150Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:11:07.0874504Z               )
2022-11-23T02:11:07.0874784Z               (multihead_attn): MultiheadAttention(
2022-11-23T02:11:07.0875194Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-11-23T02:11:07.0875545Z               )
2022-11-23T02:11:07.0875847Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-11-23T02:11:07.0876197Z               (dropout): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0876537Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-11-23T02:11:07.0876996Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0877452Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0877882Z               (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0878238Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0878571Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0878881Z               (dropout3): Dropout(p=0.1, inplace=False)
2022-11-23T02:11:07.0879224Z             )
2022-11-23T02:11:07.0879451Z           )
2022-11-23T02:11:07.0879653Z         )
2022-11-23T02:11:07.0880029Z         (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-11-23T02:11:07.0880328Z       )
2022-11-23T02:11:07.0880522Z     )
2022-11-23T02:11:07.0880819Z     (output_proj): Linear(in_features=16, out_features=23, bias=True)
2022-11-23T02:11:07.0881314Z     (bn): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
2022-11-23T02:11:07.0881637Z   )
2022-11-23T02:11:07.0881828Z )
2022-11-23T02:11:07.0882218Z ERROR: expected to be in states [<TrainingState.IDLE: 1>] but current state is TrainingState.SUMMON_FULL_PARAMS
2022-11-23T02:11:07.0882597Z ok (4.955s)
2022-11-23T02:11:07.0882865Z   test_nested_module_apply (__main__.TestApply)
2022-11-23T02:11:07.0883483Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45994
2022-11-23T02:11:07.0884025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45995
2022-11-23T02:11:07.0884616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:07.0885067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:07.0885641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:07.0886115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:07.0886741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:07.0887207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:07.0887785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:07.0888251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:07.0888694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:11:07.0889192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:11:07.0889847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:07.0890519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:07.0891049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:07.0891919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:07.0893222Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:11:07.0894019Z   warnings.warn(
2022-11-23T02:11:07.0895146Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:11:07.0895913Z   warnings.warn(
2022-11-23T02:11:07.0896162Z dist init r=0, world=2
2022-11-23T02:11:07.0896416Z dist init r=1, world=2
2022-11-23T02:11:07.0896750Z ok (3.410s)
2022-11-23T02:11:07.0897046Z   test_transformer_module_apply (__main__.TestApply)
2022-11-23T02:11:07.0897691Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46073
2022-11-23T02:11:07.0898214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46074
2022-11-23T02:11:07.0898824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:07.0899281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:07.0899860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:07.0900316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:07.0900896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:07.0901344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:07.0901897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:07.0902359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:07.0902812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:11:07.0903309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:11:07.0904025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:07.0904732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:07.0905255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:07.0905733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:07.0906980Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:11:07.0907753Z   warnings.warn(
2022-11-23T02:11:07.0908902Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:11:07.0909672Z   warnings.warn(
2022-11-23T02:11:07.0909922Z dist init r=0, world=2
2022-11-23T02:11:07.0910154Z dist init r=1, world=2
2022-11-23T02:11:07.0910389Z ok (3.810s)
2022-11-23T02:11:07.0910539Z 
2022-11-23T02:11:07.0910809Z ----------------------------------------------------------------------
2022-11-23T02:11:07.0911123Z Ran 3 tests in 12.175s
2022-11-23T02:11:07.0911283Z 
2022-11-23T02:11:07.0911396Z OK
2022-11-23T02:11:07.0911640Z 
2022-11-23T02:11:07.0911852Z Generating XML reports...
2022-11-23T02:11:07.0912469Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123021054.xml
2022-11-23T02:11:07.0912803Z 
2022-11-23T02:11:07.0913144Z ##[endgroup]
2022-11-23T02:11:07.0913742Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_hyiv6a2g)
2022-11-23T02:11:07.0914187Z 
2022-11-23T02:11:07.0914470Z Running distributed/_shard/test_partial_tensor ... [2022-11-23 02:11:07.082340]
2022-11-23T02:11:07.0915148Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:11:07.082703]
2022-11-23T02:11:23.0992898Z 
2022-11-23T02:11:23.0993471Z Expand the folded group to see the log file of distributed/_shard/test_partial_tensor
2022-11-23T02:11:23.0994704Z ##[group]PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_od9wf3re)
2022-11-23T02:11:23.0995083Z 
2022-11-23T02:11:23.0995205Z Running tests...
2022-11-23T02:11:23.0996073Z ----------------------------------------------------------------------
2022-11-23T02:11:23.0996904Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor
2022-11-23T02:11:23.0997428Z   test_cat (__main__.TestPartialTensorOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:11:23.0997896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46187
2022-11-23T02:11:23.0998329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46188
2022-11-23T02:11:23.0999034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46189
2022-11-23T02:11:23.0999523Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46190
2022-11-23T02:11:23.1000397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1000869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1001461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1001935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1002506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1002959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1003533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1004003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1004565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1005015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1005644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1006121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1006704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1007130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1007704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1008166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1008589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:23.1009069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:23.1009527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:23.1009995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:23.1010373Z skip: Need at least 4 CUDA devices (4.068s)
2022-11-23T02:11:23.1010974Z   test_cat_errors (__main__.TestPartialTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46323
2022-11-23T02:11:23.1011809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46324
2022-11-23T02:11:23.1012263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46325
2022-11-23T02:11:23.1012709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46326
2022-11-23T02:11:23.1013373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1013830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1014379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1014830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1015407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1015883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1016442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1016882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1017449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1018004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1018612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1019079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1019654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1020087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1020660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1021121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1021542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:23.1022017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:23.1022478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:23.1022943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:23.1023317Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:11:23.1023788Z   test_transpose (__main__.TestPartialTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46459
2022-11-23T02:11:23.1024314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46460
2022-11-23T02:11:23.1024745Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46461
2022-11-23T02:11:23.1025185Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46462
2022-11-23T02:11:23.1025786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1026237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1026794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1027263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1027837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1028373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1028931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1029397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1029970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1030395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1030968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1031428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1032002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1032428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1032997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1033461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1033882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:23.1034356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:23.1034872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:23.1035352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:23.1035727Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:11:23.1036222Z   test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46595
2022-11-23T02:11:23.1036774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46596
2022-11-23T02:11:23.1037223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46597
2022-11-23T02:11:23.1037646Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46598
2022-11-23T02:11:23.1038252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1038706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1039266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1039737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1040321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1040765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1041328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1041789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1042363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1042785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1043357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1043823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1044395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1044818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1045465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1045929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1046364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:23.1046820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:23.1047279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:23.1047751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:23.1048131Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:11:23.1048637Z   test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46731
2022-11-23T02:11:23.1049186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46732
2022-11-23T02:11:23.1049636Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46733
2022-11-23T02:11:23.1050060Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46734
2022-11-23T02:11:23.1050664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1051114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1051962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1052454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1053043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1053492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1054051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1054515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1055090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1055532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1056085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1056549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1057122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:23.1057544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:23.1058112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:23.1058575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:23.1059010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:23.1059470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:23.1059931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:23.1060400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:23.1060775Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:11:23.1060970Z 
2022-11-23T02:11:23.1061244Z ----------------------------------------------------------------------
2022-11-23T02:11:23.1061578Z Ran 5 tests in 13.710s
2022-11-23T02:11:23.1061743Z 
2022-11-23T02:11:23.1061855Z OK (skipped=5)
2022-11-23T02:11:23.1062075Z 
2022-11-23T02:11:23.1062204Z Generating XML reports...
2022-11-23T02:11:23.1062824Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20221123021108.xml
2022-11-23T02:11:23.1075344Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20221123021108.xml
2022-11-23T02:11:23.1075764Z 
2022-11-23T02:11:23.1076112Z ##[endgroup]
2022-11-23T02:11:23.1076761Z FINISHED PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_od9wf3re)
2022-11-23T02:11:23.1077154Z 
2022-11-23T02:11:23.1077426Z Running distributed/_tensor/test_api ... [2022-11-23 02:11:23.099382]
2022-11-23T02:11:23.1078126Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:11:23.099666]
2022-11-23T02:11:39.2067231Z 
2022-11-23T02:11:39.2067862Z Expand the folded group to see the log file of distributed/_tensor/test_api
2022-11-23T02:11:39.2068926Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_o_rcpi3g)
2022-11-23T02:11:39.2069562Z 
2022-11-23T02:11:39.2069691Z Running tests...
2022-11-23T02:11:39.2070234Z ----------------------------------------------------------------------
2022-11-23T02:11:39.2071082Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_api
2022-11-23T02:11:39.2071900Z   test_distribute_module (__main__.DTensorAPITest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:11:39.2072672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46902
2022-11-23T02:11:39.2073115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46903
2022-11-23T02:11:39.2073813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46904
2022-11-23T02:11:39.2074327Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46905
2022-11-23T02:11:39.2075198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2075710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2076408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2077071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2077710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2078396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2078985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2079739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2080321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2081067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2081655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2082396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2083117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2083753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2084436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2085042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2085737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:39.2086255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:39.2086976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:39.2087456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:39.2088065Z skip: Need at least 4 CUDA devices (3.997s)
2022-11-23T02:11:39.2088622Z   test_distribute_module_input_fn_output_fn (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47038
2022-11-23T02:11:39.2089330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47039
2022-11-23T02:11:39.2089891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47040
2022-11-23T02:11:39.2090342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47041
2022-11-23T02:11:39.2091258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2092162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2092881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2093434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2094358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2094952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2095709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2096223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2097057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2097488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2098357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2098830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2099689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2100122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2100988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2101468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2102052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:39.2102672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:39.2103140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:39.2103888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:39.2104271Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:11:39.2104945Z   test_distribute_tensor (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47174
2022-11-23T02:11:39.2105542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47175
2022-11-23T02:11:39.2106048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47176
2022-11-23T02:11:39.2106711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47177
2022-11-23T02:11:39.2107540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2108214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2108788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2109544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2110139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2110889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2111462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2112201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2112799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2113431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2114149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2114773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2115492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2116082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2116868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2117372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2118032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:39.2118524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:39.2119260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:39.2119747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:39.2120152Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:11:39.2120876Z   test_distribute_tensor_errors (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47310
2022-11-23T02:11:39.2121416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47311
2022-11-23T02:11:39.2122153Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47312
2022-11-23T02:11:39.2122590Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47313
2022-11-23T02:11:39.2123488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2123962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2124750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2125287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2126039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2126630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2127273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2127961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2128571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2129354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2129945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2130698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2131289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2135003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2135627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2136377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2136827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:39.2137291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:39.2138045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:39.2138526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:39.2138908Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:11:39.2139404Z   test_distribute_tensor_uneven_sharding (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47446
2022-11-23T02:11:39.2140233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47447
2022-11-23T02:11:39.2140805Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47448
2022-11-23T02:11:39.2141232Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47449
2022-11-23T02:11:39.2141851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2142315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2142880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2143355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2143944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2144698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2145274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2145750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2146334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2146781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2147340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2147808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2148390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:39.2148816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:39.2149394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:39.2149860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:39.2150300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:11:39.2150761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:39.2151372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:39.2152114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:11:39.2152500Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:11:39.2152705Z 
2022-11-23T02:11:39.2152992Z ----------------------------------------------------------------------
2022-11-23T02:11:39.2153330Z Ran 5 tests in 13.740s
2022-11-23T02:11:39.2153499Z 
2022-11-23T02:11:39.2153612Z OK (skipped=5)
2022-11-23T02:11:39.2153750Z 
2022-11-23T02:11:39.2153883Z Generating XML reports...
2022-11-23T02:11:39.2154461Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20221123021125.xml
2022-11-23T02:11:39.2154796Z 
2022-11-23T02:11:39.2155128Z ##[endgroup]
2022-11-23T02:11:39.2155684Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_o_rcpi3g)
2022-11-23T02:11:39.2156027Z 
2022-11-23T02:11:39.2156316Z Running distributed/test_c10d_object_collectives ... [2022-11-23 02:11:39.206785]
2022-11-23T02:11:39.2157022Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:11:39.207049]
2022-11-23T02:11:55.9414727Z 
2022-11-23T02:11:55.9415244Z Expand the folded group to see the log file of distributed/test_c10d_object_collectives
2022-11-23T02:11:55.9416515Z ##[group]PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_9iqy_qm6)
2022-11-23T02:11:55.9416923Z 
2022-11-23T02:11:55.9417041Z Running tests...
2022-11-23T02:11:55.9417581Z ----------------------------------------------------------------------
2022-11-23T02:11:55.9418149Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives
2022-11-23T02:11:55.9418686Z   test_all_gather_object (__main__.TestObjectCollectives) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:11:55.9419221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47617
2022-11-23T02:11:55.9419678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47618
2022-11-23T02:11:55.9420305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:55.9420745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:55.9421337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:55.9421810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:55.9422375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:55.9422826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:55.9423407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:55.9423879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:55.9424309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:55.9424806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:11:55.9425298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:55.9425786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:11:55.9426433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:55.9427128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:55.9427666Z ok (4.816s)
2022-11-23T02:11:55.9428097Z   test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47696
2022-11-23T02:11:55.9428639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47697
2022-11-23T02:11:55.9429257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:55.9429719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:55.9430280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:55.9430987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:55.9431639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:55.9432099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:55.9432656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:55.9433120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:55.9433560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:55.9434041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:11:55.9434603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:55.9435111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:11:55.9435773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:55.9436459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:55.9436854Z ok (3.209s)
2022-11-23T02:11:55.9437289Z   test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47775
2022-11-23T02:11:55.9437803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47776
2022-11-23T02:11:55.9438421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:55.9438877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:55.9439450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:55.9439904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:55.9440482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:55.9440931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:55.9441508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:55.9441960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:55.9442401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:55.9442895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:11:55.9443365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:55.9443849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:11:55.9444510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:55.9445284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:55.9445661Z ok (3.210s)
2022-11-23T02:11:55.9446103Z   test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47856
2022-11-23T02:11:55.9446639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47857
2022-11-23T02:11:55.9447256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:55.9447694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:55.9448272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:55.9448742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:55.9449305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:11:55.9449748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:11:55.9450317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:11:55.9450788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:11:55.9451210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:11:55.9452172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:11:55.9452685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:11:55.9453161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:11:55.9453840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:55.9454540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:11:55.9454944Z ok (3.209s)
2022-11-23T02:11:55.9455097Z 
2022-11-23T02:11:55.9455349Z ----------------------------------------------------------------------
2022-11-23T02:11:55.9455684Z Ran 4 tests in 14.445s
2022-11-23T02:11:55.9455848Z 
2022-11-23T02:11:55.9455944Z OK
2022-11-23T02:11:55.9456080Z 
2022-11-23T02:11:55.9456188Z Generating XML reports...
2022-11-23T02:11:55.9456816Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221123021141.xml
2022-11-23T02:11:55.9457189Z 
2022-11-23T02:11:55.9457526Z ##[endgroup]
2022-11-23T02:11:55.9458137Z FINISHED PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_9iqy_qm6)
2022-11-23T02:11:55.9458511Z 
2022-11-23T02:11:55.9458798Z Running distributed/_tensor/test_tp_sharding_ops ... [2022-11-23 02:11:55.941439]
2022-11-23T02:11:55.9459517Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tp_sharding_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:11:55.941703]
2022-11-23T02:12:14.8015209Z 
2022-11-23T02:12:14.8016185Z Expand the folded group to see the log file of distributed/_tensor/test_tp_sharding_ops
2022-11-23T02:12:14.8017469Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_lragr7na)
2022-11-23T02:12:14.8017997Z 
2022-11-23T02:12:14.8018203Z Running tests...
2022-11-23T02:12:14.8018830Z ----------------------------------------------------------------------
2022-11-23T02:12:14.8019407Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops
2022-11-23T02:12:14.8020540Z   test_replicated_permute (__main__.TPShardingOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:12:14.8021024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47972
2022-11-23T02:12:14.8021466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47973
2022-11-23T02:12:14.8022104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47974
2022-11-23T02:12:14.8022573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47975
2022-11-23T02:12:14.8023205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8023667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8024455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8025402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8026132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8026693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8027891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8028619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8029324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8029798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8030376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8030847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8031415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8031868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8032436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8032884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8033326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:12:14.8033808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:14.8034284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:12:14.8034735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:14.8035122Z skip: Need at least 4 CUDA devices (4.125s)
2022-11-23T02:12:14.8035594Z   test_sharded_cat (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48108
2022-11-23T02:12:14.8036093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48109
2022-11-23T02:12:14.8036541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48110
2022-11-23T02:12:14.8036975Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48111
2022-11-23T02:12:14.8037591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8038027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8038599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8039072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8039635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8040161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8040734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8041199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8041758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8042209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8042778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8043243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8043803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8044252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8044821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8045267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8045707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:12:14.8046237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:14.8046716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:12:14.8047242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:14.8047615Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:12:14.8048089Z   test_sharded_permute (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48244
2022-11-23T02:12:14.8048617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48245
2022-11-23T02:12:14.8049047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48246
2022-11-23T02:12:14.8049490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48247
2022-11-23T02:12:14.8050098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8050552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8051108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8052111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8052730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8053186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8053739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8054210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8054783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8055211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8055789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8056255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8056832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8057390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8057967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8058429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8058850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:12:14.8059328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:14.8059795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:12:14.8060264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:14.8060642Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:12:14.8061112Z   test_sharded_split (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48380
2022-11-23T02:12:14.8061636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48381
2022-11-23T02:12:14.8062086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48382
2022-11-23T02:12:14.8062511Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48383
2022-11-23T02:12:14.8063118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8063589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8064226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8064709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8065288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8065730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8066287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8066749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8067321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8067745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8068320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8068786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8069362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8069786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8070361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8070820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8071238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:12:14.8071715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:14.8072176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:14.8072646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:12:14.8073020Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:12:14.8073495Z   test_sharded_transpose (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48516
2022-11-23T02:12:14.8074024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48517
2022-11-23T02:12:14.8074542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48518
2022-11-23T02:12:14.8074964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48519
2022-11-23T02:12:14.8075569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8076019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8076580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8077052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8077624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8078067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8078616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8079059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8079632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8080083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8080667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8081197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8081794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8082219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8082787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8083255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8083695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:14.8084152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:14.8084613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:12:14.8085083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:12:14.8085464Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:12:14.8085931Z   test_sharded_view (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48652
2022-11-23T02:12:14.8086446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48653
2022-11-23T02:12:14.8086898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48654
2022-11-23T02:12:14.8087323Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48655
2022-11-23T02:12:14.8087924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8088370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8089046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8089762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8090351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8090793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8091344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8092121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8092703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8093126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8093682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:14.8094128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:14.8094705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8095160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8095742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:14.8096210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:14.8096647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:14.8097106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:12:14.8097565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:12:14.8098032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:14.8098496Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:12:14.8098705Z 
2022-11-23T02:12:14.8098980Z ----------------------------------------------------------------------
2022-11-23T02:12:14.8099316Z Ran 6 tests in 16.475s
2022-11-23T02:12:14.8099480Z 
2022-11-23T02:12:14.8099589Z OK (skipped=6)
2022-11-23T02:12:14.8099727Z 
2022-11-23T02:12:14.8099886Z Generating XML reports...
2022-11-23T02:12:14.8101070Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20221123021157.xml
2022-11-23T02:12:14.8101496Z 
2022-11-23T02:12:14.8101834Z ##[endgroup]
2022-11-23T02:12:14.8102446Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_lragr7na)
2022-11-23T02:12:14.8102814Z 
2022-11-23T02:12:14.8103125Z Running distributed/checkpoint/test_file_system_checkpoint ... [2022-11-23 02:12:14.801679]
2022-11-23T02:12:14.8103869Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:12:14.801945]
2022-11-23T02:12:36.8782323Z 
2022-11-23T02:12:36.8783010Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint
2022-11-23T02:12:36.8784058Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_nsv6be53)
2022-11-23T02:12:36.8784505Z 
2022-11-23T02:12:36.8786359Z Running tests...
2022-11-23T02:12:36.8787107Z ----------------------------------------------------------------------
2022-11-23T02:12:36.8788004Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint
2022-11-23T02:12:36.8788810Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:12:36.8791732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48823
2022-11-23T02:12:36.8792308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48824
2022-11-23T02:12:36.8793005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8793475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8794306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8794783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8795373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8795809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8796413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8796888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8797332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:36.8797811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:12:36.8798309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:36.8798804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:12:36.8799467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8800141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8800549Z ok (4.963s)
2022-11-23T02:12:36.8801145Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48906
2022-11-23T02:12:36.8801736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48907
2022-11-23T02:12:36.8802340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8802804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8803384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8803861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8804427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8804880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8805464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8805938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8806364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:36.8806843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:36.8807332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:12:36.8807816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:12:36.8808477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8809175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8809578Z ok (3.912s)
2022-11-23T02:12:36.8810423Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48989
2022-11-23T02:12:36.8810996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48990
2022-11-23T02:12:36.8812068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8812664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8813247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8813723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8814337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8814792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8815348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8815819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8816261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:36.8816755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:12:36.8817281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:36.8817772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:12:36.8818436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8819230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8819621Z ok (3.410s)
2022-11-23T02:12:36.8820103Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49072
2022-11-23T02:12:36.8820680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49073
2022-11-23T02:12:36.8821284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8821736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8822311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8822777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8823346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8823796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8824366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8824814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8825250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:36.8825727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:36.8826213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:12:36.8826691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:12:36.8827349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8828038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8828431Z ok (4.112s)
2022-11-23T02:12:36.8828777Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.045s)
2022-11-23T02:12:36.8829426Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49155
2022-11-23T02:12:36.8830126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49156
2022-11-23T02:12:36.8830721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8831176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8831748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8832222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8832783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:12:36.8833232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:12:36.8833797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:12:36.8834268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:12:36.8834688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:12:36.8835179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:12:36.8835661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:12:36.8836180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:12:36.8836855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8837543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:12:36.8837937Z ok (3.310s)
2022-11-23T02:12:36.8838076Z 
2022-11-23T02:12:36.8838348Z ----------------------------------------------------------------------
2022-11-23T02:12:36.8838682Z Ran 6 tests in 19.753s
2022-11-23T02:12:36.8838847Z 
2022-11-23T02:12:36.8838940Z OK
2022-11-23T02:12:36.8839074Z 
2022-11-23T02:12:36.8839182Z Generating XML reports...
2022-11-23T02:12:36.8839869Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123021216.xml
2022-11-23T02:12:36.8840805Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123021216.xml
2022-11-23T02:12:36.8841808Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123021216.xml
2022-11-23T02:12:36.8842299Z 
2022-11-23T02:12:36.8842728Z ##[endgroup]
2022-11-23T02:12:36.8843416Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_nsv6be53)
2022-11-23T02:12:36.8843829Z 
2022-11-23T02:12:36.8844115Z Running distributed/checkpoint/test_checkpoint ... [2022-11-23 02:12:36.878249]
2022-11-23T02:12:36.8844823Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:12:36.878527]
2022-11-23T02:13:01.9101795Z 
2022-11-23T02:13:01.9102546Z Expand the folded group to see the log file of distributed/checkpoint/test_checkpoint
2022-11-23T02:13:01.9105611Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_wqpqtjxe)
2022-11-23T02:13:01.9106036Z 
2022-11-23T02:13:01.9106139Z Running tests...
2022-11-23T02:13:01.9106763Z ----------------------------------------------------------------------
2022-11-23T02:13:01.9107816Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_checkpoint
2022-11-23T02:13:01.9108386Z   test_default_metadata (__main__.TestDistributedCheckpointing) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:13:01.9108879Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49273
2022-11-23T02:13:01.9109339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49274
2022-11-23T02:13:01.9109982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9112637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9113421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9113936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9114539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9114993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9115586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9116063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9116515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:01.9117152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:01.9117737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:01.9118499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:01.9119187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:01.9119882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:01.9120295Z ok (4.830s)
2022-11-23T02:13:01.9120799Z   test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49352
2022-11-23T02:13:01.9121464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49353
2022-11-23T02:13:01.9122094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9122533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9123118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9123591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9124178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9124606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9125183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9125649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9126073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:01.9126570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:01.9127064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:01.9127558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:01.9128311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:01.9129004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:01.9129408Z ok (3.310s)
2022-11-23T02:13:01.9129854Z   test_dummy_reader_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49431
2022-11-23T02:13:01.9130372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49432
2022-11-23T02:13:01.9130827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49433
2022-11-23T02:13:01.9131283Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49434
2022-11-23T02:13:01.9132262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9132730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9133315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9133793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9134358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9134813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9135486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9135975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9136542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9136993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9137838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9138299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9138881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9139327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9139906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9140356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9140801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:01.9141278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:01.9141731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:13:01.9142204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:13:01.9142600Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:13:01.9143082Z   test_dummy_writer_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49567
2022-11-23T02:13:01.9143595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49568
2022-11-23T02:13:01.9144049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49569
2022-11-23T02:13:01.9144496Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49570
2022-11-23T02:13:01.9145091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9145548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9146241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9146720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9147287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9147738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9148310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9148785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9149347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9149795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9150369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9150823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9151407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9151853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9152423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9152929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9153387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:13:01.9153867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:13:01.9154323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:01.9154801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:01.9155190Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:13:01.9155680Z   test_load_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49703
2022-11-23T02:13:01.9156203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49704
2022-11-23T02:13:01.9156654Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49705
2022-11-23T02:13:01.9157099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49706
2022-11-23T02:13:01.9157717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9158151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9158728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9159205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9159767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9160218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9160792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9161265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9161825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9162277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9162851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9163374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9163954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9164401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9164977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9165421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9165870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:01.9166350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:13:01.9166825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:13:01.9167277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:01.9167675Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:13:01.9168173Z   test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49839
2022-11-23T02:13:01.9168701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49840
2022-11-23T02:13:01.9169149Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49841
2022-11-23T02:13:01.9169649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49842
2022-11-23T02:13:01.9170278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9170714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9171290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9172154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9172727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9173179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9173753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9174223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9174790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9175242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9175808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9176261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9176819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9177301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9177867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9178337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9178791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:01.9179247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:13:01.9179708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:13:01.9180174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:01.9180609Z ok (2.411s)
2022-11-23T02:13:01.9181052Z   test_save_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49975
2022-11-23T02:13:01.9181582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49976
2022-11-23T02:13:01.9182033Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49977
2022-11-23T02:13:01.9182472Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49978
2022-11-23T02:13:01.9183088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9183523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9184084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9184533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9185101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9185568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9186151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9186598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9187259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9187728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9188291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9188767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9189354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9189811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9190361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9190827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9191305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:13:01.9191788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:13:01.9192241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:01.9192705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:01.9193094Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:13:01.9193594Z   test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50111
2022-11-23T02:13:01.9194125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50112
2022-11-23T02:13:01.9194576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50113
2022-11-23T02:13:01.9195023Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50114
2022-11-23T02:13:01.9195636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9196082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9196661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9197136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9197828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9198274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9198824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9199292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9199869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9200303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9200871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9201339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9201941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:01.9202375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:01.9202941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:01.9203407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:01.9203846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:13:01.9204357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:13:01.9204831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:01.9205297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:01.9205628Z ok (2.410s)
2022-11-23T02:13:01.9205781Z 
2022-11-23T02:13:01.9206056Z ----------------------------------------------------------------------
2022-11-23T02:13:01.9206398Z Ran 8 tests in 22.703s
2022-11-23T02:13:01.9206562Z 
2022-11-23T02:13:01.9206673Z OK (skipped=4)
2022-11-23T02:13:01.9206809Z 
2022-11-23T02:13:01.9206933Z Generating XML reports...
2022-11-23T02:13:01.9207597Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123021238.xml
2022-11-23T02:13:01.9208451Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123021238.xml
2022-11-23T02:13:01.9208829Z 
2022-11-23T02:13:01.9209265Z ##[endgroup]
2022-11-23T02:13:01.9209902Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_wqpqtjxe)
2022-11-23T02:13:01.9210280Z 
2022-11-23T02:13:01.9210557Z Running distributed/_tensor/test_redistribute ... [2022-11-23 02:13:01.910463]
2022-11-23T02:13:01.9211253Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:13:01.910746]
2022-11-23T02:13:28.6281883Z 
2022-11-23T02:13:28.6284568Z Expand the folded group to see the log file of distributed/_tensor/test_redistribute
2022-11-23T02:13:28.6285628Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_f_ut8zo6)
2022-11-23T02:13:28.6286112Z 
2022-11-23T02:13:28.6286299Z Running tests...
2022-11-23T02:13:28.6287094Z ----------------------------------------------------------------------
2022-11-23T02:13:28.6288058Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_redistribute
2022-11-23T02:13:28.6288577Z   test_multi_dim_mesh (__main__.MultiDimRedistributeTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:13:28.6289472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50282
2022-11-23T02:13:28.6290230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50283
2022-11-23T02:13:28.6290677Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50284
2022-11-23T02:13:28.6292189Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50285
2022-11-23T02:13:28.6292620Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 50286
2022-11-23T02:13:28.6293063Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 50287
2022-11-23T02:13:28.6293509Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 50288
2022-11-23T02:13:28.6293926Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 50289
2022-11-23T02:13:28.6294571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6303500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6304291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6304798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6305386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6305817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6306566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6307068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6307669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6308110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6308690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6309171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6309751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6310266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6310851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6311309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6311889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6312334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6312891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6313363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6313940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6314382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6314937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6315407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6315985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6316408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6316979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6317552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6318128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6318552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6319121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6319638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6320090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:13:28.6320553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:13:28.6321016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:13:28.6321485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:28.6321939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:13:28.6322398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:28.6322866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:13:28.6323326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:13:28.6323698Z skip: Need at least 8 CUDA devices (4.214s)
2022-11-23T02:13:28.6324261Z   test_partial_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50554
2022-11-23T02:13:28.6324819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50555
2022-11-23T02:13:28.6325421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6325875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6326453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6326923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6327487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6327936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6328513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6328981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6329402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:28.6329898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:28.6330390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:28.6330854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:28.6331974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6332737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6333139Z ok (3.310s)
2022-11-23T02:13:28.6333550Z   test_partial_to_shard (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50637
2022-11-23T02:13:28.6334067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50638
2022-11-23T02:13:28.6334676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6335226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6335808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6336277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6336850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6337282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6337853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6338312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6338747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:28.6339225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:28.6339715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:28.6340200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:28.6340835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6341606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6342009Z ok (3.310s)
2022-11-23T02:13:28.6342426Z   test_replicate_to_partial (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50716
2022-11-23T02:13:28.6342951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50717
2022-11-23T02:13:28.6343576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6344013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6344597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6345064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6345626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6346074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6346642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6347111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6347529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:28.6348023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:28.6348506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:28.6348986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:28.6349619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6350305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6350834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:13:28.6351329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:13:28.6351986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:13:28.6352559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:13:28.6353209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:13:28.6353735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:13:28.6354371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:13:28.6355059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:13:28.6355586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T02:13:28.6356080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T02:13:28.6356714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:13:28.6357393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:13:28.6357789Z ok (3.511s)
2022-11-23T02:13:28.6358249Z   test_replicate_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50814
2022-11-23T02:13:28.6358836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50815
2022-11-23T02:13:28.6359463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6359916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6360497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6360958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6361542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6361991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6362546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6363025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6363468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:28.6363965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:28.6364436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:28.6364929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:28.6365590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6366279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6366659Z ok (3.310s)
2022-11-23T02:13:28.6367114Z   test_replicate_to_shard_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50897
2022-11-23T02:13:28.6367659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50898
2022-11-23T02:13:28.6368255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6368708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6369353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6369831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6370393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6370842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6372058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6372573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6373021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:28.6373521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:28.6374010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:28.6374484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:28.6375161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6375852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6376251Z ok (3.311s)
2022-11-23T02:13:28.6376783Z   test_shard_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50980
2022-11-23T02:13:28.6377350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50981
2022-11-23T02:13:28.6377965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6378408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6378985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6379458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6380042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:28.6380471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:28.6381051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:28.6381522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:28.6381947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:28.6382439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:28.6382931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:28.6383421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:28.6384058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6384748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:28.6385148Z ok (3.410s)
2022-11-23T02:13:28.6385303Z 
2022-11-23T02:13:28.6385579Z ----------------------------------------------------------------------
2022-11-23T02:13:28.6385898Z Ran 7 tests in 24.376s
2022-11-23T02:13:28.6386066Z 
2022-11-23T02:13:28.6386180Z OK (skipped=1)
2022-11-23T02:13:28.6386340Z 
2022-11-23T02:13:28.6386467Z Generating XML reports...
2022-11-23T02:13:28.6387041Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20221123021303.xml
2022-11-23T02:13:28.6387923Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20221123021303.xml
2022-11-23T02:13:28.6388300Z 
2022-11-23T02:13:28.6388746Z ##[endgroup]
2022-11-23T02:13:28.6389368Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_f_ut8zo6)
2022-11-23T02:13:28.6389730Z 
2022-11-23T02:13:28.6389981Z Running distributed/fsdp/test_fsdp_meta ... [2022-11-23 02:13:28.628453]
2022-11-23T02:13:28.6390658Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:13:28.628737]
2022-11-23T02:13:58.7039305Z 
2022-11-23T02:13:58.7044176Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta
2022-11-23T02:13:58.7045372Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_qnvk3l2a)
2022-11-23T02:13:58.7045742Z 
2022-11-23T02:13:58.7045839Z Running tests...
2022-11-23T02:13:58.7046359Z ----------------------------------------------------------------------
2022-11-23T02:13:58.7047412Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta
2022-11-23T02:13:58.7048387Z   test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:13:58.7050845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51100
2022-11-23T02:13:58.7052232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51101
2022-11-23T02:13:58.7053289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7053979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7054976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7055742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7056767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7057238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7057814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7058292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7058753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:58.7059256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:58.7059931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7060634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7061152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:58.7061630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:58.7061978Z dist init r=0, world=2
2022-11-23T02:13:58.7062235Z dist init r=1, world=2
2022-11-23T02:13:58.7062487Z ok (4.876s)
2022-11-23T02:13:58.7062944Z   test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:13:58.7063636Z   test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51179
2022-11-23T02:13:58.7070337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51180
2022-11-23T02:13:58.7071043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7071540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7072129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7072610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7073197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7073647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7074233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7074696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7075157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:58.7075665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:58.7076333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7077194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7077757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:58.7078243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:58.7078612Z dist init r=0, world=2
2022-11-23T02:13:58.7078856Z dist init r=1, world=2
2022-11-23T02:13:58.7079108Z ok (3.810s)
2022-11-23T02:13:58.7079603Z   test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51262
2022-11-23T02:13:58.7080171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51263
2022-11-23T02:13:58.7080799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7081264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7081831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7082307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7082890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7083338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7083902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7084427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7084888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:58.7085386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:58.7086051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7086727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7087256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:58.7087732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:58.7088170Z dist init r=1, world=2
2022-11-23T02:13:58.7088426Z dist init r=0, world=2
2022-11-23T02:13:58.7088669Z ok (3.812s)
2022-11-23T02:13:58.7089167Z   test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51345
2022-11-23T02:13:58.7089733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51346
2022-11-23T02:13:58.7090358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7090811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7091372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7092300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7092898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7093343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7093902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7094368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7094826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:58.7095430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:58.7096100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7096790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7097314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:58.7097769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:58.7098129Z dist init r=1, world=2
2022-11-23T02:13:58.7098386Z dist init r=0, world=2
2022-11-23T02:13:58.7098626Z ok (3.812s)
2022-11-23T02:13:58.7099104Z   test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51428
2022-11-23T02:13:58.7099691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51429
2022-11-23T02:13:58.7100306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7100741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7101320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7101795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7102376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7102806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7103376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7103843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7104277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:58.7104776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:58.7105433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7106242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7106750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:58.7107223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:58.7107574Z dist init r=1, world=2
2022-11-23T02:13:58.7107829Z dist init r=0, world=2
2022-11-23T02:13:58.7108053Z ok (3.810s)
2022-11-23T02:13:58.7108565Z   test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-11-23T02:13:58.7109295Z   test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:13:58.7109996Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-11-23T02:13:58.7110699Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-11-23T02:13:58.7111392Z   test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51511
2022-11-23T02:13:58.7112036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51512
2022-11-23T02:13:58.7112676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7113114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7113692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7114168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7114733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7115183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7115759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7116230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7116668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:58.7117170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:58.7117829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7118503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7119029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:58.7119503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:58.7119867Z dist init r=1, world=2
2022-11-23T02:13:58.7120104Z dist init r=0, world=2
2022-11-23T02:13:58.7120347Z ok (3.810s)
2022-11-23T02:13:58.7120828Z   test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51594
2022-11-23T02:13:58.7121436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51595
2022-11-23T02:13:58.7122049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7122583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7123162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7123616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7124194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:13:58.7124644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:13:58.7125220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:13:58.7125671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:13:58.7126126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:13:58.7126630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:13:58.7127275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7127963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:13:58.7128484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:13:58.7129025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:13:58.7129383Z dist init r=0, world=2
2022-11-23T02:13:58.7129639Z dist init r=1, world=2
2022-11-23T02:13:58.7129880Z ok (3.810s)
2022-11-23T02:13:58.7130349Z   test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:13:58.7131040Z   test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:13:58.7131400Z 
2022-11-23T02:13:58.7132141Z ----------------------------------------------------------------------
2022-11-23T02:13:58.7132486Z Ran 14 tests in 27.745s
2022-11-23T02:13:58.7132653Z 
2022-11-23T02:13:58.7132746Z OK (skipped=7)
2022-11-23T02:13:58.7132902Z 
2022-11-23T02:13:58.7133027Z Generating XML reports...
2022-11-23T02:13:58.7133646Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123021330.xml
2022-11-23T02:13:58.7134013Z 
2022-11-23T02:13:58.7134451Z ##[endgroup]
2022-11-23T02:13:58.7135053Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_qnvk3l2a)
2022-11-23T02:13:58.7135405Z 
2022-11-23T02:13:58.7135693Z Running distributed/fsdp/test_fsdp_flatten_params ... [2022-11-23 02:13:58.704077]
2022-11-23T02:13:58.7136412Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:13:58.704384]
2022-11-23T02:14:31.8476653Z 
2022-11-23T02:14:31.8477404Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_flatten_params
2022-11-23T02:14:31.8481355Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_n7hcr_6a)
2022-11-23T02:14:31.8481773Z 
2022-11-23T02:14:31.8481909Z Running tests...
2022-11-23T02:14:31.8482419Z ----------------------------------------------------------------------
2022-11-23T02:14:31.8483006Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params
2022-11-23T02:14:31.8483460Z   test_empty_module (__main__.TestFlattenParams)
2022-11-23T02:14:31.8483918Z Tests flattening an empty module (i.e. one without any parameters). ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:14:31.8484650Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51712
2022-11-23T02:14:31.8488558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:31.8489044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:31.8489664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:31.8490160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:31.8490631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:31.8491291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:14:31.8492156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:31.8492532Z dist init r=0, world=1
2022-11-23T02:14:31.8492764Z ok (4.753s)
2022-11-23T02:14:31.8493091Z   test_flat_param_shard_metadata (__main__.TestFlattenParams)
2022-11-23T02:14:31.8493626Z Tests that ``FlatParameter`` shard metadata are computed as expected. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51752
2022-11-23T02:14:31.8494348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:31.8494957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:31.8495573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:31.8496055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:31.8496498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:31.8497175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:14:31.8497709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:31.8498079Z dist init r=0, world=1
2022-11-23T02:14:31.8498307Z ok (3.010s)
2022-11-23T02:14:31.8498606Z   test_flatten_nothing (__main__.TestFlattenParams)
2022-11-23T02:14:31.8499120Z Tests that constructing a ``FlatParamHandle`` with no parameters ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51792
2022-11-23T02:14:31.8499802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:31.8500262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:31.8500885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:31.8501368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:31.8501831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:31.8502478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:14:31.8503008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:31.8503369Z dist init r=0, world=1
2022-11-23T02:14:31.8503596Z ok (3.007s)
2022-11-23T02:14:31.8503915Z   test_numel_with_shared_params (__main__.TestFlattenParams)
2022-11-23T02:14:31.8504441Z Tests that numel is preserved after flattening when there are shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51832
2022-11-23T02:14:31.8505144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:31.8505691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:31.8506280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:31.8506754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:31.8507217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:31.8507858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:14:31.8508389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:31.8508750Z dist init r=0, world=1
2022-11-23T02:14:31.8508980Z ok (3.007s)
2022-11-23T02:14:31.8509300Z   test_numel_without_shared_params (__main__.TestFlattenParams)
2022-11-23T02:14:31.8509833Z Tests that numel is preserved after flattening when there are no shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51872
2022-11-23T02:14:31.8510544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:31.8510974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:31.8511552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:31.8512024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:31.8512523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:31.8513199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:14:31.8513728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:31.8514093Z dist init r=0, world=1
2022-11-23T02:14:31.8514326Z ok (3.007s)
2022-11-23T02:14:31.8514645Z   test_output_with_shared_params (__main__.TestFlattenParams)
2022-11-23T02:14:31.8515177Z Tests a forward pass after flattening when there are shared parameters ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51912
2022-11-23T02:14:31.8515865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:31.8516324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:31.8516903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:31.8517379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:31.8517820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:31.8518481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:14:31.8519014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:31.8519358Z dist init r=0, world=1
2022-11-23T02:14:31.8519608Z ok (3.609s)
2022-11-23T02:14:31.8519934Z   test_output_without_shared_params (__main__.TestFlattenParams)
2022-11-23T02:14:31.8520456Z Tests a forward pass after flattening when there are no shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51952
2022-11-23T02:14:31.8521139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:31.8521595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:31.8522170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:31.8522625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:31.8523207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:31.8523877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:14:31.8524413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:31.8524758Z dist init r=0, world=1
2022-11-23T02:14:31.8525005Z ok (3.608s)
2022-11-23T02:14:31.8525315Z   test_partial_flattening (__main__.TestFlattenParams)
2022-11-23T02:14:31.8525788Z Tests flattening some submodules but not others. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51992
2022-11-23T02:14:31.8526461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:31.8526913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:31.8527494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:31.8527949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:31.8528410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:31.8529074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:14:31.8529603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:31.8530912Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:14:31.8532406Z   warnings.warn(
2022-11-23T02:14:31.8532672Z dist init r=0, world=1
2022-11-23T02:14:31.8532919Z ok (3.107s)
2022-11-23T02:14:31.8533236Z   test_pnorm_after_step_with_shared_params (__main__.TestFlattenParams)
2022-11-23T02:14:31.8533783Z Tests for parameter Frobenius norm parity after an optimizer step when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52032
2022-11-23T02:14:31.8534507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:14:31.8534966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:14:31.8535530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:14:31.8536007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:14:31.8536470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:14:31.8537136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:14:31.8537647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:14:31.8538010Z dist init r=0, world=1
2022-11-23T02:14:31.8538259Z ok (3.710s)
2022-11-23T02:14:31.8538411Z 
2022-11-23T02:14:31.8538683Z ----------------------------------------------------------------------
2022-11-23T02:14:31.8539026Z Ran 9 tests in 30.820s
2022-11-23T02:14:31.8539190Z 
2022-11-23T02:14:31.8539288Z OK
2022-11-23T02:14:31.8539425Z 
2022-11-23T02:14:31.8539551Z Generating XML reports...
2022-11-23T02:14:31.8540152Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20221123021400.xml
2022-11-23T02:14:31.8540515Z 
2022-11-23T02:14:31.8541032Z ##[endgroup]
2022-11-23T02:14:31.8541678Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_n7hcr_6a)
2022-11-23T02:14:31.8542059Z 
2022-11-23T02:14:31.8542340Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-11-23 02:14:31.847756]
2022-11-23T02:14:31.8543069Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:14:31.848045]
2022-11-23T02:15:07.0361416Z 
2022-11-23T02:15:07.0362177Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-11-23T02:15:07.0363162Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_xpdnvsxo)
2022-11-23T02:15:07.0364123Z 
2022-11-23T02:15:07.0367495Z Running tests...
2022-11-23T02:15:07.0368075Z ----------------------------------------------------------------------
2022-11-23T02:15:07.0368696Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-11-23T02:15:07.0369223Z   test_grad_scaling (__main__.TestShardGradScaler) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:15:07.0369575Z ok (1.650s)
2022-11-23T02:15:07.0369922Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... ok (0.002s)
2022-11-23T02:15:07.0370363Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... ok (0.006s)
2022-11-23T02:15:07.0371241Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52124
2022-11-23T02:15:07.0374118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52125
2022-11-23T02:15:07.0374835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0375321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0375909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0376378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0376963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0377426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0378006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0378464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0378929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:07.0379436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:07.0380106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0380813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0381335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:07.0381820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:07.0382304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0382772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0384060Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0385122Z   warnings.warn(
2022-11-23T02:15:07.0386303Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0387077Z   warnings.warn(
2022-11-23T02:15:07.0387460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0387942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0388315Z dist init r=1, world=2
2022-11-23T02:15:07.0388574Z dist init r=0, world=2
2022-11-23T02:15:07.0388799Z ok (3.913s)
2022-11-23T02:15:07.0389325Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52207
2022-11-23T02:15:07.0390045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52208
2022-11-23T02:15:07.0390695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0391139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0391717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0392198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0392780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0393207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0393779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0394246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0394691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:07.0395195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:07.0395856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0396550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0397058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:07.0397535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:07.0398013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0398500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0399760Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0400622Z   warnings.warn(
2022-11-23T02:15:07.0401780Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0402543Z   warnings.warn(
2022-11-23T02:15:07.0402918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0403385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0403757Z dist init r=1, world=2
2022-11-23T02:15:07.0404011Z dist init r=0, world=2
2022-11-23T02:15:07.0404238Z ok (3.811s)
2022-11-23T02:15:07.0404792Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52290
2022-11-23T02:15:07.0405442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52291
2022-11-23T02:15:07.0406058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0406493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0407132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0407619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0408185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0408857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0409436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0409896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0426388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:07.0426904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:07.0427616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0428318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0428844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:07.0429308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:07.0429786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0430273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0431557Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0432328Z   warnings.warn(
2022-11-23T02:15:07.0433474Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0434429Z   warnings.warn(
2022-11-23T02:15:07.0434804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0435292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0435640Z dist init r=0, world=2
2022-11-23T02:15:07.0435895Z dist init r=1, world=2
2022-11-23T02:15:07.0436134Z ok (3.911s)
2022-11-23T02:15:07.0436652Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52373
2022-11-23T02:15:07.0437285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52374
2022-11-23T02:15:07.0437910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0438368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0438931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0439407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0440092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0440541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0441120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0441583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0442042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:07.0442526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:07.0443185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0443872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0444400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:07.0444858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:07.0445335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0445820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0447077Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0447854Z   warnings.warn(
2022-11-23T02:15:07.0449009Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0449857Z   warnings.warn(
2022-11-23T02:15:07.0450231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0450699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0451061Z dist init r=1, world=2
2022-11-23T02:15:07.0451315Z dist init r=0, world=2
2022-11-23T02:15:07.0452145Z ok (3.911s)
2022-11-23T02:15:07.0452708Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52456
2022-11-23T02:15:07.0453336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52457
2022-11-23T02:15:07.0453969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0454409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0454991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0455462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0456037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0456464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0457138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0457620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0458057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:07.0458562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:07.0459232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0459920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0460428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:07.0460907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:07.0461390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0461875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0463129Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0463921Z   warnings.warn(
2022-11-23T02:15:07.0465067Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0465837Z   warnings.warn(
2022-11-23T02:15:07.0466109Z   File "<string>", line 1, in <module>
2022-11-23T02:15:07.0466464Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:15:07.0466933Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:15:07.0467313Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:15:07.0467670Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:15:07.0468059Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:15:07.0468394Z     self.run()
2022-11-23T02:15:07.0468711Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:15:07.0469076Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:15:07.0469606Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:15:07.0469996Z     self.run_test(test_name, pipe)
2022-11-23T02:15:07.0470511Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:15:07.0470908Z     getattr(self, test_name)()
2022-11-23T02:15:07.0471427Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:15:07.0471780Z     fn()
2022-11-23T02:15:07.0472268Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:15:07.0472665Z     test(self, **param_kwargs)
2022-11-23T02:15:07.0473176Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:15:07.0473547Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0474051Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:15:07.0474479Z     self._test_fsdp_parity(
2022-11-23T02:15:07.0474987Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:15:07.0475406Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:15:07.0475967Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:15:07.0476359Z     output = model(*input)
2022-11-23T02:15:07.0476816Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:15:07.0477202Z     return forward_call(*input, **kwargs)
2022-11-23T02:15:07.0477747Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:15:07.0478194Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:15:07.0478760Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:15:07.0479155Z     _lazy_init(state, module)
2022-11-23T02:15:07.0479661Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:15:07.0480057Z     handle.init_flat_param_attributes()
2022-11-23T02:15:07.0480572Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:15:07.0480955Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0481475Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:15:07.0481858Z     p_assert(
2022-11-23T02:15:07.0482327Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:15:07.0482694Z     traceback.print_stack()
2022-11-23T02:15:07.0482980Z   File "<string>", line 1, in <module>
2022-11-23T02:15:07.0483354Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:15:07.0483726Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:15:07.0484078Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:15:07.0484528Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:15:07.0484919Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:15:07.0485238Z     self.run()
2022-11-23T02:15:07.0485571Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:15:07.0485938Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:15:07.0486441Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:15:07.0486826Z     self.run_test(test_name, pipe)
2022-11-23T02:15:07.0487353Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:15:07.0487746Z     getattr(self, test_name)()
2022-11-23T02:15:07.0488242Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:15:07.0488611Z     fn()
2022-11-23T02:15:07.0489106Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:15:07.0489483Z     test(self, **param_kwargs)
2022-11-23T02:15:07.0489992Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:15:07.0490382Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0490825Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:15:07.0491283Z     self._test_fsdp_parity(
2022-11-23T02:15:07.0492400Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:15:07.0492833Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:15:07.0493372Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:15:07.0493774Z     output = model(*input)
2022-11-23T02:15:07.0494247Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:15:07.0494632Z     return forward_call(*input, **kwargs)
2022-11-23T02:15:07.0495161Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:15:07.0495617Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:15:07.0496184Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:15:07.0496559Z     _lazy_init(state, module)
2022-11-23T02:15:07.0497063Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:15:07.0497469Z     handle.init_flat_param_attributes()
2022-11-23T02:15:07.0497979Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:15:07.0498351Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0498890Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:15:07.0499267Z     p_assert(
2022-11-23T02:15:07.0499723Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:15:07.0500106Z     traceback.print_stack()
2022-11-23T02:15:07.0500497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0500974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0501462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0501947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0502303Z dist init r=0, world=2
2022-11-23T02:15:07.0502878Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T02:15:07.0503325Z dist init r=1, world=2
2022-11-23T02:15:07.0503790Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T02:15:07.0504203Z ok (3.911s)
2022-11-23T02:15:07.0504726Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52539
2022-11-23T02:15:07.0505335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52540
2022-11-23T02:15:07.0505954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0506390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0506976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0507451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0508029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0508456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0509105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0509582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0510022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:07.0510524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:07.0511193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0511876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0512382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:07.0512855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:07.0513338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0513822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0515078Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0515854Z   warnings.warn(
2022-11-23T02:15:07.0516994Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0517767Z   warnings.warn(
2022-11-23T02:15:07.0518035Z   File "<string>", line 1, in <module>
2022-11-23T02:15:07.0518391Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:15:07.0518833Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:15:07.0519206Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:15:07.0519561Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:15:07.0519952Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:15:07.0520289Z     self.run()
2022-11-23T02:15:07.0520606Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:15:07.0520977Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:15:07.0521497Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:15:07.0521886Z     self.run_test(test_name, pipe)
2022-11-23T02:15:07.0522395Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:15:07.0522792Z     getattr(self, test_name)()
2022-11-23T02:15:07.0523308Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:15:07.0523665Z     fn()
2022-11-23T02:15:07.0524156Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:15:07.0524549Z     test(self, **param_kwargs)
2022-11-23T02:15:07.0525128Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:15:07.0525506Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0526014Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:15:07.0526443Z     self._test_fsdp_parity(
2022-11-23T02:15:07.0526948Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:15:07.0527368Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:15:07.0527926Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:15:07.0528325Z     output = model(*input)
2022-11-23T02:15:07.0528782Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:15:07.0529174Z     return forward_call(*input, **kwargs)
2022-11-23T02:15:07.0529718Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:15:07.0530164Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:15:07.0530489Z   File "<string>", line 1, in <module>
2022-11-23T02:15:07.0531017Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:15:07.0531410Z     _lazy_init(state, module)
2022-11-23T02:15:07.0532429Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:15:07.0532849Z     handle.init_flat_param_attributes()
2022-11-23T02:15:07.0533234Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:15:07.0533587Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:15:07.0534100Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:15:07.0534481Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0534821Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:15:07.0535198Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:15:07.0535756Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:15:07.0536145Z     p_assert(
2022-11-23T02:15:07.0536470Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:15:07.0536924Z     self.run()
2022-11-23T02:15:07.0537405Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:15:07.0537770Z     traceback.print_stack()
2022-11-23T02:15:07.0538131Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:15:07.0538500Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:15:07.0538996Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:15:07.0539382Z     self.run_test(test_name, pipe)
2022-11-23T02:15:07.0539912Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:15:07.0540304Z     getattr(self, test_name)()
2022-11-23T02:15:07.0540800Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:15:07.0541172Z     fn()
2022-11-23T02:15:07.0541658Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:15:07.0542035Z     test(self, **param_kwargs)
2022-11-23T02:15:07.0542547Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:15:07.0542939Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0543382Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:15:07.0543783Z     self._test_fsdp_parity(
2022-11-23T02:15:07.0544382Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:15:07.0544818Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:15:07.0545359Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:15:07.0545756Z     output = model(*input)
2022-11-23T02:15:07.0546231Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:15:07.0546618Z     return forward_call(*input, **kwargs)
2022-11-23T02:15:07.0547145Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:15:07.0547602Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:15:07.0548166Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:15:07.0548547Z     _lazy_init(state, module)
2022-11-23T02:15:07.0549055Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:15:07.0549463Z     handle.init_flat_param_attributes()
2022-11-23T02:15:07.0549978Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:15:07.0550344Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0550878Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:15:07.0551258Z     p_assert(
2022-11-23T02:15:07.0551712Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:15:07.0552092Z     traceback.print_stack()
2022-11-23T02:15:07.0552485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0552959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0553446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0553926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0554282Z dist init r=1, world=2
2022-11-23T02:15:07.0554739Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T02:15:07.0555259Z dist init r=0, world=2
2022-11-23T02:15:07.0555725Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T02:15:07.0556138Z ok (3.911s)
2022-11-23T02:15:07.0556695Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52622
2022-11-23T02:15:07.0557333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52623
2022-11-23T02:15:07.0557951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0558386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0558965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0559432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0560009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0560435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0561074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0561549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0561988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:07.0562492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:07.0563159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0563846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0564351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:07.0564825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:07.0565301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0565789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0567046Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0567830Z   warnings.warn(
2022-11-23T02:15:07.0568980Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0569740Z   warnings.warn(
2022-11-23T02:15:07.0570007Z   File "<string>", line 1, in <module>
2022-11-23T02:15:07.0570361Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:15:07.0570732Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:15:07.0571167Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:15:07.0571965Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:15:07.0572394Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:15:07.0572731Z     self.run()
2022-11-23T02:15:07.0573065Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:15:07.0573416Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:15:07.0573951Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:15:07.0574340Z     self.run_test(test_name, pipe)
2022-11-23T02:15:07.0574850Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:15:07.0575249Z     getattr(self, test_name)()
2022-11-23T02:15:07.0575770Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:15:07.0576124Z     fn()
2022-11-23T02:15:07.0576615Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:15:07.0577009Z     test(self, **param_kwargs)
2022-11-23T02:15:07.0577523Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:15:07.0577894Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0578431Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:15:07.0578863Z     self._test_fsdp_parity(
2022-11-23T02:15:07.0579370Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:15:07.0579793Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:15:07.0580351Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:15:07.0580745Z     output = model(*input)
2022-11-23T02:15:07.0581206Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:15:07.0581597Z     return forward_call(*input, **kwargs)
2022-11-23T02:15:07.0582144Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:15:07.0582584Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:15:07.0583144Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:15:07.0583539Z     _lazy_init(state, module)
2022-11-23T02:15:07.0584041Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:15:07.0584436Z     handle.init_flat_param_attributes()
2022-11-23T02:15:07.0584944Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:15:07.0585324Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0585839Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:15:07.0586221Z     p_assert(
2022-11-23T02:15:07.0586687Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:15:07.0587071Z     traceback.print_stack()
2022-11-23T02:15:07.0587345Z   File "<string>", line 1, in <module>
2022-11-23T02:15:07.0587715Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:15:07.0588088Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:15:07.0588443Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:15:07.0588814Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:15:07.0589302Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:15:07.0589621Z     self.run()
2022-11-23T02:15:07.0589954Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:15:07.0590317Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:15:07.0590820Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:15:07.0591208Z     self.run_test(test_name, pipe)
2022-11-23T02:15:07.0591735Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:15:07.0592126Z     getattr(self, test_name)()
2022-11-23T02:15:07.0592623Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:15:07.0592993Z     fn()
2022-11-23T02:15:07.0593483Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:15:07.0593863Z     test(self, **param_kwargs)
2022-11-23T02:15:07.0594372Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:15:07.0594763Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0595206Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:15:07.0595605Z     self._test_fsdp_parity(
2022-11-23T02:15:07.0596184Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:15:07.0596618Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:15:07.0597158Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:15:07.0597554Z     output = model(*input)
2022-11-23T02:15:07.0598036Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:15:07.0598418Z     return forward_call(*input, **kwargs)
2022-11-23T02:15:07.0598944Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:15:07.0599399Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:15:07.0599962Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:15:07.0600341Z     _lazy_init(state, module)
2022-11-23T02:15:07.0600841Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:15:07.0601244Z     handle.init_flat_param_attributes()
2022-11-23T02:15:07.0601756Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:15:07.0602122Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0602655Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:15:07.0603038Z     p_assert(
2022-11-23T02:15:07.0603490Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:15:07.0603870Z     traceback.print_stack()
2022-11-23T02:15:07.0604263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0604737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0605222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0605704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0606067Z dist init r=1, world=2
2022-11-23T02:15:07.0606526Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T02:15:07.0607052Z dist init r=0, world=2
2022-11-23T02:15:07.0607517Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T02:15:07.0607941Z ok (4.011s)
2022-11-23T02:15:07.0608458Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52705
2022-11-23T02:15:07.0609074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52706
2022-11-23T02:15:07.0609693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0610132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0610710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0611182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0612281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:07.0612717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:07.0613295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:07.0613855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:07.0614325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:07.0614812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:07.0615477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0616174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:07.0616682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:07.0617153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:07.0617625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0618115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0619369Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0620159Z   warnings.warn(
2022-11-23T02:15:07.0621303Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:15:07.0622076Z   warnings.warn(
2022-11-23T02:15:07.0622345Z   File "<string>", line 1, in <module>
2022-11-23T02:15:07.0622615Z   File "<string>", line 1, in <module>
2022-11-23T02:15:07.0622984Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:15:07.0623449Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:15:07.0623811Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T02:15:07.0624180Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T02:15:07.0624550Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:15:07.0624970Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:15:07.0625334Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T02:15:07.0625705Z     return self._bootstrap(parent_sentinel)
2022-11-23T02:15:07.0626098Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:15:07.0626418Z     self.run()
2022-11-23T02:15:07.0626758Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T02:15:07.0627091Z     self.run()
2022-11-23T02:15:07.0627407Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:15:07.0627782Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:15:07.0628155Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T02:15:07.0628501Z     self._target(*self._args, **self._kwargs)
2022-11-23T02:15:07.0629027Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:15:07.0629425Z     self.run_test(test_name, pipe)
2022-11-23T02:15:07.0629925Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T02:15:07.0630359Z     self.run_test(test_name, pipe)
2022-11-23T02:15:07.0630898Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:15:07.0631295Z     getattr(self, test_name)()
2022-11-23T02:15:07.0631794Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T02:15:07.0632191Z     getattr(self, test_name)()
2022-11-23T02:15:07.0632706Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:15:07.0633074Z     fn()
2022-11-23T02:15:07.0633535Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T02:15:07.0633893Z     fn()
2022-11-23T02:15:07.0634380Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:15:07.0634763Z     test(self, **param_kwargs)
2022-11-23T02:15:07.0635275Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T02:15:07.0635673Z     test(self, **param_kwargs)
2022-11-23T02:15:07.0636180Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:15:07.0636553Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0637070Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T02:15:07.0637462Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0637887Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:15:07.0638304Z     self._test_fsdp_parity(
2022-11-23T02:15:07.0638741Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler
2022-11-23T02:15:07.0639160Z     self._test_fsdp_parity(
2022-11-23T02:15:07.0639666Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:15:07.0640090Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:15:07.0640625Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T02:15:07.0641101Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T02:15:07.0641655Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:15:07.0642047Z     output = model(*input)
2022-11-23T02:15:07.0642571Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T02:15:07.0642945Z     output = model(*input)
2022-11-23T02:15:07.0643419Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:15:07.0643806Z     return forward_call(*input, **kwargs)
2022-11-23T02:15:07.0644280Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T02:15:07.0644662Z     return forward_call(*input, **kwargs)
2022-11-23T02:15:07.0645204Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:15:07.0645647Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:15:07.0646220Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T02:15:07.0646676Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T02:15:07.0647238Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:15:07.0647670Z     _lazy_init(state, module)
2022-11-23T02:15:07.0648204Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T02:15:07.0648596Z     _lazy_init(state, module)
2022-11-23T02:15:07.0649083Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:15:07.0649490Z     handle.init_flat_param_attributes()
2022-11-23T02:15:07.0650020Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T02:15:07.0650421Z     handle.init_flat_param_attributes()
2022-11-23T02:15:07.0650915Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:15:07.0651298Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0652279Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T02:15:07.0652652Z     return func(*args, **kwargs)
2022-11-23T02:15:07.0653188Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:15:07.0653573Z     p_assert(
2022-11-23T02:15:07.0654085Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T02:15:07.0654456Z     p_assert(
2022-11-23T02:15:07.0654928Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:15:07.0655313Z     traceback.print_stack()
2022-11-23T02:15:07.0655788Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T02:15:07.0656162Z     traceback.print_stack()
2022-11-23T02:15:07.0656556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0657047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0657516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0657995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:15:07.0658354Z dist init r=0, world=2
2022-11-23T02:15:07.0658813Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T02:15:07.0659379Z dist init r=1, world=2
2022-11-23T02:15:07.0659850Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T02:15:07.0660280Z ok (3.811s)
2022-11-23T02:15:07.0660412Z 
2022-11-23T02:15:07.0660690Z ----------------------------------------------------------------------
2022-11-23T02:15:07.0661026Z Ran 11 tests in 32.848s
2022-11-23T02:15:07.0661189Z 
2022-11-23T02:15:07.0661283Z OK
2022-11-23T02:15:07.0661423Z 
2022-11-23T02:15:07.0661529Z Generating XML reports...
2022-11-23T02:15:07.0662158Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123021433.xml
2022-11-23T02:15:07.0663042Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123021433.xml
2022-11-23T02:15:07.0663466Z 
2022-11-23T02:15:07.0663864Z ##[endgroup]
2022-11-23T02:15:07.0664507Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_xpdnvsxo)
2022-11-23T02:15:07.0664895Z 
2022-11-23T02:15:07.0665159Z Running distributed/algorithms/test_join ... [2022-11-23 02:15:07.036626]
2022-11-23T02:15:07.0665923Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:07.036931]
2022-11-23T02:15:44.3056787Z 
2022-11-23T02:15:44.3057263Z Expand the folded group to see the log file of distributed/algorithms/test_join
2022-11-23T02:15:44.3059608Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_0nk4vbh6)
2022-11-23T02:15:44.3060286Z 
2022-11-23T02:15:44.3060445Z Running tests...
2022-11-23T02:15:44.3060950Z ----------------------------------------------------------------------
2022-11-23T02:15:44.3064333Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join
2022-11-23T02:15:44.3064802Z   test_join_kwargs (__main__.TestJoin)
2022-11-23T02:15:44.3065417Z Tests passing keyword arguments to the context manager. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:15:44.3065906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52823
2022-11-23T02:15:44.3066360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52824
2022-11-23T02:15:44.3066999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3067459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3068066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3068534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3069117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3069577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3070159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3070615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3071075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:44.3071573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:44.3072048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:44.3072801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:44.3073475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3074165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3074547Z ok (5.325s)
2022-11-23T02:15:44.3074848Z   test_multiple_joinable_disable (__main__.TestJoin)
2022-11-23T02:15:44.3075337Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52903
2022-11-23T02:15:44.3075857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52904
2022-11-23T02:15:44.3076801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3077301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3077935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3078419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3079010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3079452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3080217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3080711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3081139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:44.3081635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:44.3082132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:44.3082618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:44.3083275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3083970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3084371Z ok (3.710s)
2022-11-23T02:15:44.3084645Z   test_multiple_joinables (__main__.TestJoin)
2022-11-23T02:15:44.3085270Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52983
2022-11-23T02:15:44.3085809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52984
2022-11-23T02:15:44.3086417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3086861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3087431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3087903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3088480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3088914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3089555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3090024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3090463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:44.3091047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:44.3091926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:44.3092431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:44.3093109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3093802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3094183Z ok (3.710s)
2022-11-23T02:15:44.3094475Z   test_multiple_joinables_throw (__main__.TestJoin)
2022-11-23T02:15:44.3094955Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53063
2022-11-23T02:15:44.3095457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53064
2022-11-23T02:15:44.3096072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3096521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3097099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3097550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3098225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3098688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3099250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3099723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3100162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:44.3100652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:44.3101120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:44.3101609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:44.3102276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3102968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3103344Z ok (3.710s)
2022-11-23T02:15:44.3103620Z   test_single_joinable (__main__.TestJoin)
2022-11-23T02:15:44.3104228Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53143
2022-11-23T02:15:44.3104746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53144
2022-11-23T02:15:44.3105358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3105811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3106383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3106839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3107415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3107861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3108414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3108974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3109413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:44.3109905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:44.3110372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:44.3110859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:44.3111520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3112210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3112588Z ok (3.710s)
2022-11-23T02:15:44.3112891Z   test_single_joinable_disable (__main__.TestJoin)
2022-11-23T02:15:44.3113366Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53223
2022-11-23T02:15:44.3113864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53224
2022-11-23T02:15:44.3114471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3114921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3115552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3116017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3116597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3117043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3117605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3118075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3118514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:44.3119005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:44.3119477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:44.3119960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:44.3120618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3121304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3121681Z ok (3.710s)
2022-11-23T02:15:44.3121972Z   test_single_joinable_main_hooks (__main__.TestJoin)
2022-11-23T02:15:44.3122442Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53303
2022-11-23T02:15:44.3122936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53304
2022-11-23T02:15:44.3123547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3123995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3124568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3125024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3125600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3126113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3126673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3127195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3127639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:44.3128138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:44.3128608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:44.3129090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:44.3129744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3130437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3130812Z ok (3.730s)
2022-11-23T02:15:44.3131104Z   test_single_joinable_post_hooks (__main__.TestJoin)
2022-11-23T02:15:44.3131995Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53383
2022-11-23T02:15:44.3132508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53384
2022-11-23T02:15:44.3133205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3133667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3134241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3134694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3135278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3135724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3136277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3136741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3137186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:44.3137677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:44.3138144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:44.3138628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:44.3139293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3139981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3140353Z ok (3.710s)
2022-11-23T02:15:44.3140638Z   test_single_joinable_throw (__main__.TestJoin)
2022-11-23T02:15:44.3141108Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53463
2022-11-23T02:15:44.3141615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53464
2022-11-23T02:15:44.3142224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3142672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3143247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3143788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3144366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:15:44.3144812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:15:44.3145365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:15:44.3145835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:15:44.3146271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:15:44.3146762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:15:44.3147228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:15:44.3147715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:15:44.3148370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3149058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:15:44.3149434Z ok (3.710s)
2022-11-23T02:15:44.3149586Z 
2022-11-23T02:15:44.3149860Z ----------------------------------------------------------------------
2022-11-23T02:15:44.3150250Z Ran 9 tests in 35.028s
2022-11-23T02:15:44.3150425Z 
2022-11-23T02:15:44.3150502Z OK
2022-11-23T02:15:44.3150637Z 
2022-11-23T02:15:44.3150762Z Generating XML reports...
2022-11-23T02:15:44.3151325Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221123021508.xml
2022-11-23T02:15:44.3151655Z 
2022-11-23T02:15:44.3152109Z ##[endgroup]
2022-11-23T02:15:44.3152701Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_0nk4vbh6)
2022-11-23T02:15:44.3153062Z 
2022-11-23T02:15:44.3153325Z Running distributed/_tensor/test_dtensor ... [2022-11-23 02:15:44.305824]
2022-11-23T02:15:44.3153996Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:15:44.306131]
2022-11-23T02:16:24.5483329Z 
2022-11-23T02:16:24.5484217Z Expand the folded group to see the log file of distributed/_tensor/test_dtensor
2022-11-23T02:16:24.5485241Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_7i7al8fb)
2022-11-23T02:16:24.5485594Z 
2022-11-23T02:16:24.5485713Z Running tests...
2022-11-23T02:16:24.5486361Z ----------------------------------------------------------------------
2022-11-23T02:16:24.5489676Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_dtensor
2022-11-23T02:16:24.5490212Z   test_device_mesh_nd (__main__.DTensorMeshTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:24.5490702Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53578
2022-11-23T02:16:24.5491154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53579
2022-11-23T02:16:24.5491989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53580
2022-11-23T02:16:24.5492453Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53581
2022-11-23T02:16:24.5492896Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 53582
2022-11-23T02:16:24.5493340Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 53583
2022-11-23T02:16:24.5493753Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 53584
2022-11-23T02:16:24.5494438Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 53585
2022-11-23T02:16:24.5495091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5495535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5496108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5496553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5497142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5497598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5498188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5498671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5499268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5499695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5500271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5500740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5501424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5501876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5502450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5502920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5503486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5503943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5504519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5504986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5505560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5506009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5506580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5507057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5507623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5508075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5508646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5509091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5509667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5510262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5510829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5511273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5511713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5512268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:24.5512736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:16:24.5513186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:24.5513643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:16:24.5514106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:16:24.5514554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:16:24.5515006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5515396Z skip: Need at least 8 CUDA devices (4.128s)
2022-11-23T02:16:24.5515863Z   test_dtensor_2d_mesh (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53850
2022-11-23T02:16:24.5516362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53851
2022-11-23T02:16:24.5516808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53852
2022-11-23T02:16:24.5517252Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53853
2022-11-23T02:16:24.5517673Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 53854
2022-11-23T02:16:24.5518110Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 53855
2022-11-23T02:16:24.5518596Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 53856
2022-11-23T02:16:24.5519041Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 53857
2022-11-23T02:16:24.5519637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5520090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5520661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5521112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5521694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5522140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5522714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5523167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5523743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5524185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5524758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5525207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5525780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5526220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5526778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5527241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5527813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5528252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5528927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5529395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5529971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5530393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5530958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5531746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5532339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5532758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5533325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5533787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5534355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5534778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5535342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5535885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5536316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5536788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:24.5537256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:16:24.5537726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:16:24.5538177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:24.5538631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:16:24.5539096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:16:24.5539538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5539934Z skip: Need at least 8 CUDA devices (2.616s)
2022-11-23T02:16:24.5540431Z   test_dtensor_api_device_mesh_context_manager (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54122
2022-11-23T02:16:24.5540970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54123
2022-11-23T02:16:24.5541398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54124
2022-11-23T02:16:24.5541835Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54125
2022-11-23T02:16:24.5542273Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 54126
2022-11-23T02:16:24.5542694Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 54127
2022-11-23T02:16:24.5543129Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 54128
2022-11-23T02:16:24.5543562Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 54129
2022-11-23T02:16:24.5544173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5544610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5545184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5545740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5546321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5546748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5547317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5547779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5548344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5548789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5549357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5549813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5550371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5550815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5551384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5551830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5552460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5552922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5553491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5553942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5554521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5554963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5555520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5555948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5556515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5556986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5557550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5558014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5558588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5559030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5559583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5560045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5560481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:24.5560943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5561411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:16:24.5561866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5562327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:16:24.5562841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:16:24.5563296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:16:24.5563754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:24.5564141Z skip: Need at least 8 CUDA devices (2.715s)
2022-11-23T02:16:24.5564619Z   test_dtensor_device_mesh_device_conversion (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54394
2022-11-23T02:16:24.5565162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54395
2022-11-23T02:16:24.5565607Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54396
2022-11-23T02:16:24.5566031Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54397
2022-11-23T02:16:24.5566463Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 54398
2022-11-23T02:16:24.5566903Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 54399
2022-11-23T02:16:24.5567338Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 54400
2022-11-23T02:16:24.5567752Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 54401
2022-11-23T02:16:24.5568353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5568802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5569414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5569898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5570479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5570922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5571758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5572236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5572816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5573240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5573814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5574274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5574850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5575273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5575845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5576303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5576873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5577292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5577864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5578320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5578875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5579315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5579880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5580433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5580994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5581438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5582000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5582448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5583014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5583455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5584019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5584465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5584903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:24.5585375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:16:24.5585842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5586371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:16:24.5586846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:24.5587307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:16:24.5587748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5588204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:16:24.5588593Z skip: Need at least 8 CUDA devices (2.715s)
2022-11-23T02:16:24.5589079Z   test_dtensor_spec_local_shard_offset (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54666
2022-11-23T02:16:24.5589596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54667
2022-11-23T02:16:24.5590043Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54668
2022-11-23T02:16:24.5590485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54669
2022-11-23T02:16:24.5590908Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 54670
2022-11-23T02:16:24.5591342Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 54671
2022-11-23T02:16:24.5591772Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 54672
2022-11-23T02:16:24.5592205Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 54673
2022-11-23T02:16:24.5592801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5593248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5593818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5594270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5594850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5595289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5595859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5596305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5596974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5597414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5597982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5598427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5599001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5599440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5599988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5600445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5601023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5601458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5602006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5602464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5603028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5603499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5604088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5604550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5605120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5605546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5606113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5606571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5607138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5607561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5608123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5608580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5608997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5609473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:16:24.5609934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:16:24.5610402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:16:24.5610846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:16:24.5611304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5612104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:16:24.5612549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:16:24.5612941Z skip: Need at least 8 CUDA devices (2.615s)
2022-11-23T02:16:24.5613405Z   test_dtensor_constructor (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54938
2022-11-23T02:16:24.5614008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54939
2022-11-23T02:16:24.5614613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5615058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5615627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5616101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5616657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5617096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5617662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5618115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5618555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5619047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:24.5619537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5620002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:24.5620731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5621437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5621816Z ok (3.310s)
2022-11-23T02:16:24.5622235Z   test_dtensor_properties (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55017
2022-11-23T02:16:24.5622754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55018
2022-11-23T02:16:24.5623358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5623790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5624361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5624834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5625413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5625840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5626408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5626874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5627295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5627786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:24.5628272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5628761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:24.5629454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5630143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5630536Z ok (3.309s)
2022-11-23T02:16:24.5631014Z   test_dtensor_spec_read_only_after_set (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55096
2022-11-23T02:16:24.5631542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55097
2022-11-23T02:16:24.5632160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5632609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5633166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5633632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5634210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5634654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5635212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5635673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5636111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5636584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:24.5637076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5637632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:24.5638305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5638978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5639376Z ok (3.309s)
2022-11-23T02:16:24.5639785Z   test_dtensor_stride (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55175
2022-11-23T02:16:24.5640288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55176
2022-11-23T02:16:24.5640876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5641326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5641902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5642353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5642930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5643371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5643945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5644392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5644829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5645318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:24.5645789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5646273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:24.5646927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5647610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5648057Z ok (3.310s)
2022-11-23T02:16:24.5648456Z   test_from_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55254
2022-11-23T02:16:24.5648953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55255
2022-11-23T02:16:24.5649562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5649992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5650570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5651038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5651857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5652320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5652898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5653360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5653781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5654270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:24.5654843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5655325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:24.5655991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5656676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5657072Z ok (3.211s)
2022-11-23T02:16:24.5657473Z   test_from_local_then_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55337
2022-11-23T02:16:24.5657986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55338
2022-11-23T02:16:24.5658592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5659042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5659597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5660062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5660637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5661068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5661638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5662099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5662534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5663010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:24.5663501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5663986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:24.5664636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5665399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5665790Z ok (3.310s)
2022-11-23T02:16:24.5666188Z   test_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55420
2022-11-23T02:16:24.5666666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55421
2022-11-23T02:16:24.5667274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5667732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5668302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5668755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5669336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:24.5669784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:24.5670337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:24.5670802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:24.5671240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:24.5671794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:24.5672278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:24.5672762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:24.5673419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5674107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:24.5674483Z ok (3.310s)
2022-11-23T02:16:24.5674632Z 
2022-11-23T02:16:24.5674902Z ----------------------------------------------------------------------
2022-11-23T02:16:24.5675237Z Ran 12 tests in 37.859s
2022-11-23T02:16:24.5675404Z 
2022-11-23T02:16:24.5675496Z OK (skipped=5)
2022-11-23T02:16:24.5675648Z 
2022-11-23T02:16:24.5675772Z Generating XML reports...
2022-11-23T02:16:24.5676337Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20221123021546.xml
2022-11-23T02:16:24.5677057Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20221123021546.xml
2022-11-23T02:16:24.5677372Z 
2022-11-23T02:16:24.5677804Z ##[endgroup]
2022-11-23T02:16:24.5678396Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_7i7al8fb)
2022-11-23T02:16:24.5678743Z 
2022-11-23T02:16:24.5679006Z Running distributed/fsdp/test_fsdp_misc ... [2022-11-23 02:16:24.548491]
2022-11-23T02:16:24.5679673Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:16:24.548785]
2022-11-23T02:17:23.9383533Z 
2022-11-23T02:17:23.9384030Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_misc
2022-11-23T02:17:23.9387859Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_0082_op4)
2022-11-23T02:17:23.9388247Z 
2022-11-23T02:17:23.9388364Z Running tests...
2022-11-23T02:17:23.9388885Z ----------------------------------------------------------------------
2022-11-23T02:17:23.9389424Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc
2022-11-23T02:17:23.9390138Z   test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9390610Z Tests that passing ``sync_module_states=True`` raises an error for ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:17:23.9391109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55538
2022-11-23T02:17:23.9391546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55539
2022-11-23T02:17:23.9392177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9392648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9393232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9393698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9394279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9395200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9395795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9396258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9396719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9397342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9398061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9398736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9399273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9399749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9401993Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:17:23.9403472Z   warnings.warn(
2022-11-23T02:17:23.9405834Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:17:23.9407255Z   warnings.warn(
2022-11-23T02:17:23.9407714Z dist init r=1, world=2
2022-11-23T02:17:23.9408165Z dist init r=0, world=2
2022-11-23T02:17:23.9408554Z ok (4.944s)
2022-11-23T02:17:23.9409177Z   test_device_id_auto_wrap (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9409952Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55617
2022-11-23T02:17:23.9410495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55618
2022-11-23T02:17:23.9411104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9412187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9412976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9413436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9414017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9414467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9415039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9415496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9415956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9416458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9417104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9417800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9418326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9418800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9419138Z dist init r=0, world=2
2022-11-23T02:17:23.9419396Z dist init r=1, world=2
2022-11-23T02:17:23.9419720Z ok (3.310s)
2022-11-23T02:17:23.9420026Z   test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9420547Z Tests that passing a CPU module to FSDP preserves that the wrapped ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55696
2022-11-23T02:17:23.9421089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55697
2022-11-23T02:17:23.9421718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9422151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9422725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9423193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9423752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9424200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9424774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9425239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9425673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9426176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9426833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9427524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9428036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9428516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9428871Z dist init r=1, world=2
2022-11-23T02:17:23.9429108Z dist init r=0, world=2
2022-11-23T02:17:23.9429353Z ok (3.710s)
2022-11-23T02:17:23.9429663Z   test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9430261Z Ensures that even if device_id is specified but we have ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55779
2022-11-23T02:17:23.9430872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55780
2022-11-23T02:17:23.9431486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9431938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9432559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9433032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9433611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9434035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9434606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9435071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9435527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9436008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9436666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9437413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9437949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9438406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9438756Z dist init r=0, world=2
2022-11-23T02:17:23.9439007Z dist init r=1, world=2
2022-11-23T02:17:23.9439230Z ok (3.310s)
2022-11-23T02:17:23.9439535Z   test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9440005Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55858
2022-11-23T02:17:23.9440487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55859
2022-11-23T02:17:23.9441103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9441560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9442134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9442586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9443161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9443609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9444179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9444627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9445079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9445578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9446220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9446909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9447503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9447975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9448315Z dist init r=1, world=2
2022-11-23T02:17:23.9448567Z dist init r=0, world=2
2022-11-23T02:17:23.9448807Z ok (3.210s)
2022-11-23T02:17:23.9449095Z   test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9449561Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55937
2022-11-23T02:17:23.9450062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55938
2022-11-23T02:17:23.9450680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9451116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9452206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9452694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9453261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9453705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9454275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9454835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9455292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9455786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9456447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9457145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9457650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9458122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9458476Z dist init r=0, world=2
2022-11-23T02:17:23.9458712Z dist init r=1, world=2
2022-11-23T02:17:23.9458949Z ok (3.210s)
2022-11-23T02:17:23.9459441Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56016
2022-11-23T02:17:23.9460023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56017
2022-11-23T02:17:23.9460619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9461082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9461658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9462114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9462694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9463137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9463713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9464164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9464615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9465207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9465847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9466539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9467064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9467541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9467880Z dist init r=0, world=2
2022-11-23T02:17:23.9468130Z dist init r=1, world=2
2022-11-23T02:17:23.9468368Z ok (3.711s)
2022-11-23T02:17:23.9468871Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56099
2022-11-23T02:17:23.9469478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56100
2022-11-23T02:17:23.9470087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9470539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9471100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9471572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9472203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9472663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9473223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9473694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9474149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9474631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9475285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9475968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9476490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9476944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9477294Z dist init r=1, world=2
2022-11-23T02:17:23.9477543Z dist init r=0, world=2
2022-11-23T02:17:23.9477764Z ok (3.711s)
2022-11-23T02:17:23.9478252Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56182
2022-11-23T02:17:23.9478829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56183
2022-11-23T02:17:23.9479434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9479871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9480450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9480919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9481480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9481923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9482567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9483032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9483469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9483961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9484613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9485298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9485803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9486274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9486631Z dist init r=1, world=2
2022-11-23T02:17:23.9486866Z dist init r=0, world=2
2022-11-23T02:17:23.9487104Z ok (3.811s)
2022-11-23T02:17:23.9487617Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56265
2022-11-23T02:17:23.9488222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56266
2022-11-23T02:17:23.9488871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9489335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9489913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9490365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9490947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9491387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9492487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9492940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9493398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9493895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9494551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9495225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9495753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9496221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9496559Z dist init r=0, world=2
2022-11-23T02:17:23.9496811Z dist init r=1, world=2
2022-11-23T02:17:23.9497051Z ok (3.811s)
2022-11-23T02:17:23.9497452Z   test_fsdp_namedtuple (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56348
2022-11-23T02:17:23.9497967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56349
2022-11-23T02:17:23.9498576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9499031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9499590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9500171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9500758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9501203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9501756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9502224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9502679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9503166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9503819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9504514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9505032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9505484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9505833Z dist init r=1, world=2
2022-11-23T02:17:23.9506083Z dist init r=0, world=2
2022-11-23T02:17:23.9506306Z ok (3.310s)
2022-11-23T02:17:23.9506830Z   test_fsdp_not_all_outputs_used_in_loss (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56427
2022-11-23T02:17:23.9507379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56428
2022-11-23T02:17:23.9507994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9508436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9509011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9509477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9510035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9510483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9511058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9511517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9511953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9512447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9513106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9513789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9514293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9514764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9515648Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:17:23.9516239Z   warnings.warn(message, UserWarning)
2022-11-23T02:17:23.9517019Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:17:23.9517677Z   warnings.warn(message, UserWarning)
2022-11-23T02:17:23.9517959Z dist init r=1, world=2
2022-11-23T02:17:23.9518190Z dist init r=0, world=2
2022-11-23T02:17:23.9518431Z ok (3.913s)
2022-11-23T02:17:23.9518735Z   test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9519251Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56510
2022-11-23T02:17:23.9519774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56511
2022-11-23T02:17:23.9520386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9520832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9521399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9521868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9522444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9522889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9523498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9523974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9524425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9524919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9525569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9526257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9526780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9527236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9527595Z dist init r=0, world=2
2022-11-23T02:17:23.9527846Z dist init r=1, world=2
2022-11-23T02:17:23.9528073Z ok (3.310s)
2022-11-23T02:17:23.9528385Z   test_module_device_mismatches_device_id (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9528899Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56589
2022-11-23T02:17:23.9529429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56590
2022-11-23T02:17:23.9530025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9530478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9531052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9531998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9532640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9533091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9533663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9534112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9534682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9535182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9535842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9536518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9537043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9537518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9537878Z dist init r=1, world=2
2022-11-23T02:17:23.9538115Z dist init r=0, world=2
2022-11-23T02:17:23.9538356Z ok (3.309s)
2022-11-23T02:17:23.9538661Z   test_multi_device_not_supported (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9539291Z Tests that wrapping a multi-device module (i.e. with submodules on ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56668
2022-11-23T02:17:23.9539837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56669
2022-11-23T02:17:23.9540447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9540875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9541535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9542020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9542605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9543034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9543611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9544075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9544514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9545014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9545676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9546363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9546865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9547336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9547688Z dist init r=0, world=2
2022-11-23T02:17:23.9547939Z dist init r=1, world=2
2022-11-23T02:17:23.9548161Z ok (3.209s)
2022-11-23T02:17:23.9548430Z   test_no_params (__main__.TestFSDPMisc)
2022-11-23T02:17:23.9548903Z Test that device_id and cpu init work if module has no params ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56747
2022-11-23T02:17:23.9549414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56748
2022-11-23T02:17:23.9550022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9550471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9551028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9551497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9552158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:17:23.9552602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:17:23.9553156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:17:23.9553623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:17:23.9554077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:17:23.9554573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:17:23.9555214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9555901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:17:23.9556426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:17:23.9556895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:17:23.9557237Z dist init r=0, world=2
2022-11-23T02:17:23.9557492Z dist init r=1, world=2
2022-11-23T02:17:23.9557729Z ok (3.309s)
2022-11-23T02:17:23.9557863Z 
2022-11-23T02:17:23.9558135Z ----------------------------------------------------------------------
2022-11-23T02:17:23.9558466Z Ran 16 tests in 57.100s
2022-11-23T02:17:23.9558630Z 
2022-11-23T02:17:23.9558782Z OK
2022-11-23T02:17:23.9558928Z 
2022-11-23T02:17:23.9559037Z Generating XML reports...
2022-11-23T02:17:23.9559620Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123021626.xml
2022-11-23T02:17:23.9559953Z 
2022-11-23T02:17:23.9560287Z ##[endgroup]
2022-11-23T02:17:23.9560860Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_0082_op4)
2022-11-23T02:17:23.9561214Z 
2022-11-23T02:17:23.9561491Z Running distributed/_tensor/test_tensor_ops ... [2022-11-23 02:17:23.938536]
2022-11-23T02:17:23.9562172Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:23.938802]
2022-11-23T02:18:29.5060963Z 
2022-11-23T02:18:29.5061718Z Expand the folded group to see the log file of distributed/_tensor/test_tensor_ops
2022-11-23T02:18:29.5062666Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_qxse_dqk)
2022-11-23T02:18:29.5063035Z 
2022-11-23T02:18:29.5063155Z Running tests...
2022-11-23T02:18:29.5069418Z ----------------------------------------------------------------------
2022-11-23T02:18:29.5070023Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tensor_ops
2022-11-23T02:18:29.5070528Z   test_aten_contiguous (__main__.DistTensorOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:29.5071007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56861
2022-11-23T02:18:29.5071462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56862
2022-11-23T02:18:29.5072106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5072547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5073416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5073904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5074498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5075209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5075808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5076285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5076743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5077228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5077729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5078220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5078919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5079609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5080014Z ok (4.912s)
2022-11-23T02:18:29.5080432Z   test_clone (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56942
2022-11-23T02:18:29.5080945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56943
2022-11-23T02:18:29.5081621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5082646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5083831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5084767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5085957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5086888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5087905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5088804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5089689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5090206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5090690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5091181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5092390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5093103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5093484Z ok (3.309s)
2022-11-23T02:18:29.5093918Z   test_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57023
2022-11-23T02:18:29.5094440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57024
2022-11-23T02:18:29.5095057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5095497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5096080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5096552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5097174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5097796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5098383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5098858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5099284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5099794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5100291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5100775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5101418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5102115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5102518Z ok (3.310s)
2022-11-23T02:18:29.5102917Z   test_detach (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57106
2022-11-23T02:18:29.5103429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57107
2022-11-23T02:18:29.5104131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5104604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5105170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5105770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5106344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5106798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5107383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5107839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5108281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5108779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5109267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5109734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5110395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5111091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5111489Z ok (3.409s)
2022-11-23T02:18:29.5111891Z   test_empty_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57187
2022-11-23T02:18:29.5112403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57188
2022-11-23T02:18:29.5113015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5113447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5114029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5114499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5115166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5115599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5116173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5116634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5117057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5117557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5118046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5118533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5119176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5119867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5120267Z ok (3.309s)
2022-11-23T02:18:29.5120697Z   test_fill_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57266
2022-11-23T02:18:29.5121199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57267
2022-11-23T02:18:29.5121865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5122335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5122895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5123374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5123954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5124399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5124955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5125422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5125865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5126344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5126833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5127320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5127985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5128662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5129061Z ok (3.310s)
2022-11-23T02:18:29.5129503Z   test_fill_inplace_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57345
2022-11-23T02:18:29.5130036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57346
2022-11-23T02:18:29.5130629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5131079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5132253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5132842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5133443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5133896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5134473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5134925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5135376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5135925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5136424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5136895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5137559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5138253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5138631Z ok (3.310s)
2022-11-23T02:18:29.5139049Z   test_full_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57424
2022-11-23T02:18:29.5139644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57425
2022-11-23T02:18:29.5140282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5140720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5141296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5141770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5142329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5142777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5143354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5143822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5144243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5144736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5145224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5145712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5146353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5147044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5147444Z ok (3.310s)
2022-11-23T02:18:29.5147840Z   test_index (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57503
2022-11-23T02:18:29.5148353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57504
2022-11-23T02:18:29.5148963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5149415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5149975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5150524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5151114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5151564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5152119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5152594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5153040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5153511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5153998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5154493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5155143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5155811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5156205Z ok (11.827s)
2022-11-23T02:18:29.5156683Z   test_inplace_op (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57614
2022-11-23T02:18:29.5157194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57615
2022-11-23T02:18:29.5157810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5158263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5158845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5159300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5159875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5160318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5160892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5161341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5161783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5162274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5162744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5163235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5163893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5164581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5164959Z ok (3.310s)
2022-11-23T02:18:29.5165381Z   test_ones_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57695
2022-11-23T02:18:29.5165892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57696
2022-11-23T02:18:29.5166485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5166942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5167590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5168066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5168632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5169087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5169662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5170126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5170547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5171020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5172007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5172541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5173217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5173900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5174301Z ok (3.309s)
2022-11-23T02:18:29.5174832Z   test_ones_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57774
2022-11-23T02:18:29.5175381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57775
2022-11-23T02:18:29.5176006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5176467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5177028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5177499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5178079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5178507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5179084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5179549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5179990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5180468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5180957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5181444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5182082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5182773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5183172Z ok (3.309s)
2022-11-23T02:18:29.5183596Z   test_op_out_variant (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57853
2022-11-23T02:18:29.5184098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57854
2022-11-23T02:18:29.5184710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5185258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5185842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5186299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5186881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5187326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5187883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5188354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5188793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5189291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5189759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5190243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5190899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5191650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5192042Z ok (3.309s)
2022-11-23T02:18:29.5192467Z   test_zero_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57934
2022-11-23T02:18:29.5192986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57935
2022-11-23T02:18:29.5193583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5194040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5194615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5195085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5195645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5196096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5196667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5197114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5197552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5198048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5198536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5199001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5199655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5200350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5200752Z ok (3.309s)
2022-11-23T02:18:29.5201152Z   test_zeros_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58013
2022-11-23T02:18:29.5201662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58014
2022-11-23T02:18:29.5202371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5202806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5203388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5203862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5204445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5204881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5205457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5205923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5206344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5206838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5207329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5207817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5208453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5209205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5209618Z ok (3.309s)
2022-11-23T02:18:29.5210050Z   test_zeros_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58092
2022-11-23T02:18:29.5210560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58093
2022-11-23T02:18:29.5211177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5212148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5212729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5213207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5213792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:29.5214240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:29.5214796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:29.5215265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:29.5215706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:29.5216198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:29.5216669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:29.5217157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:29.5217824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5218498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:29.5218897Z ok (3.309s)
2022-11-23T02:18:29.5219047Z 
2022-11-23T02:18:29.5219319Z ----------------------------------------------------------------------
2022-11-23T02:18:29.5219652Z Ran 16 tests in 63.174s
2022-11-23T02:18:29.5219917Z 
2022-11-23T02:18:29.5220015Z OK
2022-11-23T02:18:29.5220153Z 
2022-11-23T02:18:29.5220284Z Generating XML reports...
2022-11-23T02:18:29.5220892Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20221123021725.xml
2022-11-23T02:18:29.5221246Z 
2022-11-23T02:18:29.5221576Z ##[endgroup]
2022-11-23T02:18:29.5222173Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_qxse_dqk)
2022-11-23T02:18:29.5222523Z 
2022-11-23T02:18:29.5222814Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-11-23 02:18:29.506292]
2022-11-23T02:18:29.5223518Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:29.506620]
2022-11-23T02:19:35.6828721Z 
2022-11-23T02:19:35.6829273Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_checkpoint
2022-11-23T02:19:35.6830361Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint__xumv2pw)
2022-11-23T02:19:35.6830742Z 
2022-11-23T02:19:35.6830921Z Running tests...
2022-11-23T02:19:35.6831440Z ----------------------------------------------------------------------
2022-11-23T02:19:35.6836704Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint
2022-11-23T02:19:35.6837646Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:19:35.6838306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58206
2022-11-23T02:19:35.6838769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58207
2022-11-23T02:19:35.6839470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6839924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6840514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6841300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6842413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6843446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6844122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6844597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6845041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6846057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6847399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6848851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6849960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6850854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6851896Z dist init r=1, world=2
2022-11-23T02:19:35.6852424Z dist init r=0, world=2
2022-11-23T02:19:35.6852948Z ok (5.564s)
2022-11-23T02:19:35.6853662Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58289
2022-11-23T02:19:35.6854585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58290
2022-11-23T02:19:35.6855229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6855688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6856273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6856737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6857319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6857768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6858326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6858801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6859255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6859750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6860399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6861178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6861728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6862212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6862556Z dist init r=0, world=2
2022-11-23T02:19:35.6862812Z dist init r=1, world=2
2022-11-23T02:19:35.6863053Z ok (3.912s)
2022-11-23T02:19:35.6863599Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58372
2022-11-23T02:19:35.6864257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58373
2022-11-23T02:19:35.6864877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6865336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6865898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6866368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6866953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6867386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6867962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6868425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6868881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6869366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6870025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6870710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6871236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6871775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6872128Z dist init r=1, world=2
2022-11-23T02:19:35.6872385Z dist init r=0, world=2
2022-11-23T02:19:35.6872613Z ok (3.912s)
2022-11-23T02:19:35.6873179Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58455
2022-11-23T02:19:35.6873829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58456
2022-11-23T02:19:35.6874447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6874879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6875455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6875931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6876495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6876942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6877511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6878030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6878480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6878982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6879640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6880333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6880841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6881315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6881670Z dist init r=1, world=2
2022-11-23T02:19:35.6881904Z dist init r=0, world=2
2022-11-23T02:19:35.6882142Z ok (4.012s)
2022-11-23T02:19:35.6882708Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58538
2022-11-23T02:19:35.6883366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58539
2022-11-23T02:19:35.6883960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6884417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6884998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6885455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6886033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6886479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6887053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6887502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6887957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6888527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6889193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6889873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6890400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6890878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6891219Z dist init r=0, world=2
2022-11-23T02:19:35.6891875Z dist init r=1, world=2
2022-11-23T02:19:35.6892225Z ok (3.812s)
2022-11-23T02:19:35.6892798Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58621
2022-11-23T02:19:35.6893444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58622
2022-11-23T02:19:35.6894067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6894520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6895081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6895661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6896265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6896718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6897278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6897750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6898201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6898698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6899346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6900039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6900567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6901025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6901381Z dist init r=0, world=2
2022-11-23T02:19:35.6901641Z dist init r=1, world=2
2022-11-23T02:19:35.6901881Z ok (3.912s)
2022-11-23T02:19:35.6902428Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58704
2022-11-23T02:19:35.6903070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58705
2022-11-23T02:19:35.6903680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6904116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6904693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6905161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6905744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6906274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6906852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6907323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6907780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6908268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6908925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6909609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6910119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6910596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6910945Z dist init r=1, world=2
2022-11-23T02:19:35.6911195Z dist init r=0, world=2
2022-11-23T02:19:35.6911418Z ok (3.912s)
2022-11-23T02:19:35.6911980Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58787
2022-11-23T02:19:35.6912689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58788
2022-11-23T02:19:35.6913302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6913758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6914333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6914802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6915364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6915812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6916382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6916851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6917288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6917784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6918442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6919121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6919648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6920120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6920478Z dist init r=1, world=2
2022-11-23T02:19:35.6920714Z dist init r=0, world=2
2022-11-23T02:19:35.6920953Z ok (4.012s)
2022-11-23T02:19:35.6921520Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58870
2022-11-23T02:19:35.6922150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58871
2022-11-23T02:19:35.6922760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6923291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6923867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6924322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6924903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6925353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6925922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6926372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6926826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6927325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6927965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6928659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6929176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6929707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6930153Z dist init r=0, world=2
2022-11-23T02:19:35.6930411Z dist init r=1, world=2
2022-11-23T02:19:35.6930635Z ok (3.811s)
2022-11-23T02:19:35.6931199Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58953
2022-11-23T02:19:35.6932398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58954
2022-11-23T02:19:35.6933015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6933475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6934055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6934535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6935099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6935546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6936122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6936596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6937034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6937536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6938202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6938879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6939457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6939936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6940293Z dist init r=1, world=2
2022-11-23T02:19:35.6940670Z dist init r=0, world=2
2022-11-23T02:19:35.6940912Z ok (3.811s)
2022-11-23T02:19:35.6941485Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59036
2022-11-23T02:19:35.6942124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59037
2022-11-23T02:19:35.6942744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6943206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6943785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6944243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6944821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6945268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6945843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6946291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6946744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6947318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6947981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6948673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6949203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6949673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6950011Z dist init r=1, world=2
2022-11-23T02:19:35.6950266Z dist init r=0, world=2
2022-11-23T02:19:35.6950509Z ok (3.811s)
2022-11-23T02:19:35.6951056Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59119
2022-11-23T02:19:35.6951707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59120
2022-11-23T02:19:35.6952320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6952775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6953337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6953813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6954397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6954848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6955400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6955865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6956318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6956795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6957451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6958213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6958735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6959198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6959554Z dist init r=0, world=2
2022-11-23T02:19:35.6959811Z dist init r=1, world=2
2022-11-23T02:19:35.6960037Z ok (3.811s)
2022-11-23T02:19:35.6960607Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59202
2022-11-23T02:19:35.6961255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59203
2022-11-23T02:19:35.6961867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6962305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6962880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6963347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6963926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6964407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6964998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6965468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6965903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6966403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6967063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6967751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6968253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6968725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6969082Z dist init r=0, world=2
2022-11-23T02:19:35.6969320Z dist init r=1, world=2
2022-11-23T02:19:35.6969562Z ok (3.912s)
2022-11-23T02:19:35.6970125Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59285
2022-11-23T02:19:35.6970776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59286
2022-11-23T02:19:35.6971374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6972318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6972911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6973391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6973958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6974400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6974968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6975540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6975999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6976505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6977169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6977851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6978380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6978848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6979190Z dist init r=0, world=2
2022-11-23T02:19:35.6979444Z dist init r=1, world=2
2022-11-23T02:19:35.6979684Z ok (3.811s)
2022-11-23T02:19:35.6980245Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59368
2022-11-23T02:19:35.6980875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59369
2022-11-23T02:19:35.6981489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6982022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6982622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6983077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6983662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6984115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6984670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6985139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6985593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6986093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6986731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6987415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6987933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6988409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6988748Z dist init r=1, world=2
2022-11-23T02:19:35.6988998Z dist init r=0, world=2
2022-11-23T02:19:35.6989235Z ok (3.911s)
2022-11-23T02:19:35.6989779Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59451
2022-11-23T02:19:35.6990431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59452
2022-11-23T02:19:35.6991041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6991494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6992048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6992595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6993177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:35.6993606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:35.6994183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:35.6994655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:35.6995107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:35.6995585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:35.6996240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6996931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:35.6997452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:35.6997905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:35.6998253Z dist init r=1, world=2
2022-11-23T02:19:35.6998506Z dist init r=0, world=2
2022-11-23T02:19:35.6998727Z ok (3.912s)
2022-11-23T02:19:35.6998874Z 
2022-11-23T02:19:35.6999209Z ----------------------------------------------------------------------
2022-11-23T02:19:35.6999558Z Ran 16 tests in 63.840s
2022-11-23T02:19:35.6999726Z 
2022-11-23T02:19:35.6999803Z OK
2022-11-23T02:19:35.6999937Z 
2022-11-23T02:19:35.7000062Z Generating XML reports...
2022-11-23T02:19:35.7000680Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123021831.xml
2022-11-23T02:19:35.7001054Z 
2022-11-23T02:19:35.7001426Z ##[endgroup]
2022-11-23T02:19:35.7002050Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint__xumv2pw)
2022-11-23T02:19:35.7002422Z 
2022-11-23T02:19:35.7002725Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-11-23 02:19:35.683122]
2022-11-23T02:19:35.7003463Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:19:35.683387]
2022-11-23T02:22:10.9164570Z 
2022-11-23T02:22:10.9165260Z Expand the folded group to see the log file of distributed/optim/test_zero_redundancy_optimizer
2022-11-23T02:22:10.9168770Z ##[group]PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_k5or6y64)
2022-11-23T02:22:10.9169599Z 
2022-11-23T02:22:10.9169762Z Running tests...
2022-11-23T02:22:10.9170300Z ----------------------------------------------------------------------
2022-11-23T02:22:10.9171216Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer
2022-11-23T02:22:10.9172161Z   test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9172686Z Check that ZeroRedundancyOptimizer properly handles adding a new ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:22:10.9173773Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.609s)
2022-11-23T02:22:10.9182374Z   test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9183243Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59569
2022-11-23T02:22:10.9183801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59570
2022-11-23T02:22:10.9184482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9184953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9185554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9186039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9186632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9187090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9187658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9188133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9188580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9189045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9189548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9190169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9190871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9191548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9191965Z ok (4.415s)
2022-11-23T02:22:10.9192481Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9193205Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59653
2022-11-23T02:22:10.9193745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59654
2022-11-23T02:22:10.9194369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9194838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9195423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9195877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9196473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9196925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9197489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9197965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9198418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9199246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9199727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9200215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9200964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9201658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9202549Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9203602Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9204288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9204785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9205250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9205733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9206090Z ok (4.111s)
2022-11-23T02:22:10.9206605Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9207355Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59766
2022-11-23T02:22:10.9207916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59767
2022-11-23T02:22:10.9208532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9208989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9209557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9210029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9210607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9211035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9212034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9212517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9212959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9213437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9213926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9214416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9215064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9215756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9216667Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9217717Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9218504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9218977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9219459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9219941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9220270Z ok (4.111s)
2022-11-23T02:22:10.9220786Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9221493Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59879
2022-11-23T02:22:10.9222044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59880
2022-11-23T02:22:10.9222651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9223107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9223682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9224156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9224720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9225237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9225826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9226277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9226718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9227200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9227692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9228174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9228834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9229529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9230438Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9231479Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9232150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9232646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9233127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9233592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9233943Z ok (4.112s)
2022-11-23T02:22:10.9234454Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9235165Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59992
2022-11-23T02:22:10.9235759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59993
2022-11-23T02:22:10.9236375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9236830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9237389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9237868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9238949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9239428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9239990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9240465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9240906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9241386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9241863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9242432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9243107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9243785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9244694Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9245749Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9246421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9246921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9247439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9247924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9248278Z ok (4.011s)
2022-11-23T02:22:10.9248777Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9249497Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60105
2022-11-23T02:22:10.9250049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60106
2022-11-23T02:22:10.9250666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9251106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9252031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9252506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9253093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9253619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9254199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9254669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9255091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9255570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9256066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9256570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9257212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9257908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9258815Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9259924Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9260590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9261079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9261559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9262049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9262383Z ok (4.112s)
2022-11-23T02:22:10.9262895Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9263603Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60218
2022-11-23T02:22:10.9264163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60219
2022-11-23T02:22:10.9264767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9265224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9265803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9266263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9266847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9267296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9267872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9268326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9268771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9269247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9269723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9270220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9270964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9271656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9272552Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9273600Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9274275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9274768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9275232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9275709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9276057Z ok (4.037s)
2022-11-23T02:22:10.9276570Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9277326Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60331
2022-11-23T02:22:10.9277886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60332
2022-11-23T02:22:10.9278499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9278959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9279517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9279991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9280574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9281004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9281582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9282051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9282492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9282951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9283443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9283941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9284605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9285281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9286195Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9287243Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9287983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9288455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9288939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9289417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9289771Z ok (4.112s)
2022-11-23T02:22:10.9290265Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9291281Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60444
2022-11-23T02:22:10.9292235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60445
2022-11-23T02:22:10.9292847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9293304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9293880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9294351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9295008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9295477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9296052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9296519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9296948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9297442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9297929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9298393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9299057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9299750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9300659Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9301689Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9302353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9302843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9303327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9303794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9304144Z ok (4.011s)
2022-11-23T02:22:10.9304659Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9305449Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60557
2022-11-23T02:22:10.9305981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60558
2022-11-23T02:22:10.9306599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9307043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9307630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9308096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9308677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9309105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9309677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9310138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9310560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9311030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9311514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9312060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9312720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9313405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9314316Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9315358Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9316008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9316496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9316976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9317451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9317782Z ok (4.011s)
2022-11-23T02:22:10.9318288Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9318985Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60670
2022-11-23T02:22:10.9319525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60671
2022-11-23T02:22:10.9320124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9320575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9321150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9321603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9322248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9322692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9323263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9323714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9324150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9324639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9325127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9325591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9326241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9326935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9327819Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9328916Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9329599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9330081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9330563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9331029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9331374Z ok (4.111s)
2022-11-23T02:22:10.9332180Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9332871Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60783
2022-11-23T02:22:10.9333424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60784
2022-11-23T02:22:10.9334050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9334507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9335067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9335540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9336120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9336558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9337110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9337575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9338014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9338467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9338952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9339537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9340199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9340870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9341776Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9342819Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9343485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9343953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9344431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9344910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9345256Z ok (4.112s)
2022-11-23T02:22:10.9345806Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9346526Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60896
2022-11-23T02:22:10.9347073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60897
2022-11-23T02:22:10.9347743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9348182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9348759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9349225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9349783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9350234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9350802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9351269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9351689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9352160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9352645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9353121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9353772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9354459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9355365Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9356404Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9357120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9357610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9358089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9358558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9358910Z ok (4.112s)
2022-11-23T02:22:10.9359417Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9360118Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61009
2022-11-23T02:22:10.9360651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61010
2022-11-23T02:22:10.9361262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9361715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9362290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9362793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9363383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9363829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9364388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9364857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9365293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9365764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9366235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9366728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9367386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9368077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9368967Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9370016Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9370679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9371162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9372001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9372489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9372837Z ok (4.112s)
2022-11-23T02:22:10.9373323Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9374118Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61122
2022-11-23T02:22:10.9374660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61123
2022-11-23T02:22:10.9375284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9375720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9376296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9376766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9377343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9377776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9378347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9378807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9379242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9379697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9380246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9380760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9381406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9382097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9383004Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9384052Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9384718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9385189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9385663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9386138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9386470Z ok (4.013s)
2022-11-23T02:22:10.9386972Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9387670Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61235
2022-11-23T02:22:10.9388211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61236
2022-11-23T02:22:10.9388807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9389258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9389830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9390379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9390943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9391387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9391959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9392409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9392851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9393321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9393811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9394291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9394948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9395637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9396545Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9397628Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9398309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9398795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9399277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9399736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9400085Z ok (4.112s)
2022-11-23T02:22:10.9400585Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9401276Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61348
2022-11-23T02:22:10.9401820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61349
2022-11-23T02:22:10.9402433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9402887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9403452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9403917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9404492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9404935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9405490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9405950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9406384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9406858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9407405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9407888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9408542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9409213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9410126Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9411166Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-11-23T02:22:10.9412163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9412634Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9413113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9413593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9413940Z ok (4.112s)
2022-11-23T02:22:10.9414435Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9415082Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61461
2022-11-23T02:22:10.9415616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61462
2022-11-23T02:22:10.9416232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9416675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9417248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9417713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9418280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9418734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9419301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9419762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9420183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9420673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9421157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9421622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9422276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9422963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9423516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12obxbzg
2022-11-23T02:22:10.9424041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12obxbzg/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9424570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzywmjp1g
2022-11-23T02:22:10.9425194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzywmjp1g/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9426260Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9427836Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9428961Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9429763Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9430517Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9431239Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9431980Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9432719Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9433458Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9434162Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9434880Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9435613Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9436342Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9437073Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9437780Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9438508Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9439301Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9440032Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9440478Z ok (4.514s)
2022-11-23T02:22:10.9440907Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9441528Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61544
2022-11-23T02:22:10.9442060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61545
2022-11-23T02:22:10.9442977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9443451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9444036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9444512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9445071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9445587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9446175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9446626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9447065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9447600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9448090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9448565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9449223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9449922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9450473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmc4n67nq
2022-11-23T02:22:10.9451000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmc4n67nq/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9451872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4iwwhk9z
2022-11-23T02:22:10.9452424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4iwwhk9z/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9453480Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9454984Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9456214Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9456952Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9457697Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9458436Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9459165Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9459878Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9460665Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9461420Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9462153Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9462888Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9463600Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9464333Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9465062Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9465792Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9466521Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9467228Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9467695Z ok (4.414s)
2022-11-23T02:22:10.9468122Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9468729Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61627
2022-11-23T02:22:10.9469266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61628
2022-11-23T02:22:10.9469892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9470407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9470970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9471440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9472021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9472464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9473018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9473479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9473919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9474391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9474882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9475365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9476021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9476740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9477305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjpnftql_
2022-11-23T02:22:10.9477847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjpnftql_/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9478378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptaur0v0s
2022-11-23T02:22:10.9478901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptaur0v0s/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9479959Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9481526Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9482651Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9483396Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9484140Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9484859Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9485658Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9486394Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9487133Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9487868Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9488580Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9489312Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9490034Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9490762Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9491869Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9492609Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9493343Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9494073Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9494542Z ok (4.515s)
2022-11-23T02:22:10.9494955Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9495580Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61710
2022-11-23T02:22:10.9496115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61711
2022-11-23T02:22:10.9496736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9497173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9516089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9516641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9517265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9517713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9518302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9518776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9519209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9519682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9520311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9520817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9521474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9522164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9522723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_csbm5h
2022-11-23T02:22:10.9523267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_csbm5h/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9523785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpko09oijq
2022-11-23T02:22:10.9524325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpko09oijq/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9525446Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9526982Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9528116Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9528869Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9529598Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9530342Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9531092Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9532084Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9532814Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9533529Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9534256Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9534978Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9535818Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9536522Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9537252Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9537989Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9538725Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9539459Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9539907Z ok (4.414s)
2022-11-23T02:22:10.9540343Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9541039Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61793
2022-11-23T02:22:10.9541582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61794
2022-11-23T02:22:10.9542213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9542669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9543255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9543719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9544298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9544744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9545319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9545771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9546213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9546684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9547163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9547719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9548383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9549076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9549615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56msjt14
2022-11-23T02:22:10.9550159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56msjt14/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9550693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya0rys6p
2022-11-23T02:22:10.9551234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya0rys6p/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9552353Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9553888Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9555006Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9555763Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9556547Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9557291Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9558029Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9558747Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9559487Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9560234Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9560967Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9561682Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9562420Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9563152Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9563895Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9564630Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9565351Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9566150Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9566614Z ok (4.415s)
2022-11-23T02:22:10.9567040Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9567651Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61876
2022-11-23T02:22:10.9568193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61877
2022-11-23T02:22:10.9568814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9569268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9569836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9570303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9570879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9571304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9572241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9572717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9573159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9573634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9574128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9574613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9575259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9575953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9576511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxt4d2q_
2022-11-23T02:22:10.9577052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxt4d2q_/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9577569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbld6lnd
2022-11-23T02:22:10.9578110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbld6lnd/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9579182Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9580716Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:22:10.9581910Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9582657Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9583408Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9584133Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9584878Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9585621Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9586355Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9587120Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9587864Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9588607Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9589346Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9590074Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9590793Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9591529Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9592261Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9592992Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-11-23T02:22:10.9593442Z ok (4.441s)
2022-11-23T02:22:10.9593789Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9594338Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61959
2022-11-23T02:22:10.9594869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61960
2022-11-23T02:22:10.9595480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9595931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9596510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9597034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9597624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9598068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9598637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9599093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9599534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9600011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9600504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9600995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9601657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9602348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9602727Z ok (4.012s)
2022-11-23T02:22:10.9603102Z   test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9603741Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62043
2022-11-23T02:22:10.9604303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62044
2022-11-23T02:22:10.9604901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9605356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9605932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9606402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9606964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9607408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9607981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9608426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9608866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9609361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9609851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9610322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9610985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9611915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9612304Z ok (4.514s)
2022-11-23T02:22:10.9612684Z   test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9613420Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62127
2022-11-23T02:22:10.9613978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62128
2022-11-23T02:22:10.9614681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9615134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9615712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9616183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9616747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9617192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9617764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9618211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9618658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9619190Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-11-23T02:22:10.9619703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9620211Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-11-23T02:22:10.9620597Z ok (2.510s)
2022-11-23T02:22:10.9621016Z   test_sharding (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9622194Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-11-23T02:22:10.9622999Z   test_step (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9623561Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62195
2022-11-23T02:22:10.9624111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62196
2022-11-23T02:22:10.9624724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9625161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9625738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9626210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9626788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9627218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9627792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9628258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9628682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9629161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9629651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9630155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9630799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9631566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9631959Z ok (3.912s)
2022-11-23T02:22:10.9632324Z   test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9632886Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62278
2022-11-23T02:22:10.9633428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62279
2022-11-23T02:22:10.9634043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9634478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9635062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9635533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9636114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9636540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9637109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9637572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9638044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9638551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9639038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9639530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9640180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9640868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9641264Z ok (3.912s)
2022-11-23T02:22:10.9641621Z   test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9642168Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62361
2022-11-23T02:22:10.9642702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62362
2022-11-23T02:22:10.9643311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9643746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9644323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9644791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9645369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9645799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9646371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9646840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9647318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9647798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9648283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9648847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9649497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9650184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9650739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0kkr8qi
2022-11-23T02:22:10.9651289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0kkr8qi/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9651992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmg__x9a
2022-11-23T02:22:10.9652528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmg__x9a/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9653046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9653518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9654179Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmph0kkr8qi'>
2022-11-23T02:22:10.9654639Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:22:10.9655215Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpdmg__x9a'>
2022-11-23T02:22:10.9655725Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:22:10.9656025Z ok (2.608s)
2022-11-23T02:22:10.9656382Z   test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9656922Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62439
2022-11-23T02:22:10.9657459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62440
2022-11-23T02:22:10.9658079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9658529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9659093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9659563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9660146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9660593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9661147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9661614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9662059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9662535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:10.9663027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9663515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9664181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9664856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:10.9665410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1s5u9kc
2022-11-23T02:22:10.9665954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1s5u9kc/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9666570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnf2dwof
2022-11-23T02:22:10.9667094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnf2dwof/_remote_module_non_scriptable.py
2022-11-23T02:22:10.9667692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9668165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:10.9668827Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpd1s5u9kc'>
2022-11-23T02:22:10.9669288Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:22:10.9669852Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpmnf2dwof'>
2022-11-23T02:22:10.9670305Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:22:10.9670596Z ok (4.512s)
2022-11-23T02:22:10.9671016Z   test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9671736Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62523
2022-11-23T02:22:10.9672273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62524
2022-11-23T02:22:10.9672882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9673370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9673963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9674433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9675013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9675446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9676015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9676482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9676922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9677384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9677778Z skip: Need at least 4 CUDA devices (2.509s)
2022-11-23T02:22:10.9678238Z   test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-11-23T02:22:10.9678954Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62591
2022-11-23T02:22:10.9679495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62592
2022-11-23T02:22:10.9680100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9680551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9681110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9681581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9682161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9682586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9683157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9683619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9684123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9684582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.9684969Z skip: Need at least 4 CUDA devices (2.508s)
2022-11-23T02:22:10.9685368Z   test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-11-23T02:22:10.9685932Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62659
2022-11-23T02:22:10.9686645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9687097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9687674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9688130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9688568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9689061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9689722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:22:10.9690103Z ok (2.408s)
2022-11-23T02:22:10.9690508Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-11-23T02:22:10.9691066Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62694
2022-11-23T02:22:10.9691985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9692442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9693025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9693492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9693917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9694409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9695073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:22:10.9695468Z ok (3.209s)
2022-11-23T02:22:10.9695818Z   test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-11-23T02:22:10.9696409Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62736
2022-11-23T02:22:10.9697124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9697559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9698134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9698605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9699046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9699525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9700184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:22:10.9700582Z ok (2.407s)
2022-11-23T02:22:10.9700914Z   test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-11-23T02:22:10.9701580Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62771
2022-11-23T02:22:10.9702287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9702738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9703297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9703770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9704210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9704684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9705345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:22:10.9705747Z ok (3.311s)
2022-11-23T02:22:10.9706120Z   test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-11-23T02:22:10.9706694Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62813
2022-11-23T02:22:10.9707402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9707852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9708506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9708974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9709417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9709914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9710567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:22:10.9710964Z ok (3.308s)
2022-11-23T02:22:10.9711325Z   test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-11-23T02:22:10.9711889Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62855
2022-11-23T02:22:10.9712565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9713016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9713598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9714069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9714499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9714993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9715653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:22:10.9716033Z ok (3.209s)
2022-11-23T02:22:10.9716400Z   test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-11-23T02:22:10.9716967Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62897
2022-11-23T02:22:10.9717652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9718086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9718730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9719206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9719630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9720119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9720777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:22:10.9721174Z ok (3.308s)
2022-11-23T02:22:10.9721502Z   test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-11-23T02:22:10.9722036Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62939
2022-11-23T02:22:10.9722713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.9723172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.9723732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.9724203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.9724646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.9725173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:10.9725843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:22:10.9726239Z ok (2.408s)
2022-11-23T02:22:10.9726388Z 
2022-11-23T02:22:10.9726657Z ----------------------------------------------------------------------
2022-11-23T02:22:10.9726975Z Ran 42 tests in 152.621s
2022-11-23T02:22:10.9727144Z 
2022-11-23T02:22:10.9727252Z OK (skipped=4)
2022-11-23T02:22:10.9727405Z 
2022-11-23T02:22:10.9727529Z Generating XML reports...
2022-11-23T02:22:10.9728237Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221123021937.xml
2022-11-23T02:22:10.9729224Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221123021937.xml
2022-11-23T02:22:10.9729663Z 
2022-11-23T02:22:10.9730057Z ##[endgroup]
2022-11-23T02:22:10.9730737Z FINISHED PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_k5or6y64)
2022-11-23T02:22:10.9731116Z 
2022-11-23T02:22:10.9731401Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-11-23 02:22:10.917528]
2022-11-23T02:22:10.9732386Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:10.917835]
2022-11-23T02:25:46.9912902Z 
2022-11-23T02:25:46.9913591Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_optim_state
2022-11-23T02:25:46.9918227Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_q6gsm50a)
2022-11-23T02:25:46.9918623Z 
2022-11-23T02:25:46.9918932Z Running tests...
2022-11-23T02:25:46.9919528Z ----------------------------------------------------------------------
2022-11-23T02:25:46.9920130Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state
2022-11-23T02:25:46.9920628Z   test_flatten_sharded_optim_state_dict_nested (__main__.TestFSDPOptimState)
2022-11-23T02:25:46.9921212Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:46.9921981Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63011
2022-11-23T02:25:46.9922443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63012
2022-11-23T02:25:46.9923090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9923533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9924126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9924652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9925232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9925692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9926273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9926747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9927196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:46.9927695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:46.9928366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9929174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9929706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:46.9930192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:46.9931083Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:46.9932050Z   warnings.warn(
2022-11-23T02:25:46.9932844Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:46.9933373Z   warnings.warn(
2022-11-23T02:25:46.9934183Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:46.9934742Z   warnings.warn(
2022-11-23T02:25:46.9935539Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:46.9936072Z   warnings.warn(
2022-11-23T02:25:46.9936332Z dist init r=1, world=2
2022-11-23T02:25:46.9936589Z dist init r=0, world=2
2022-11-23T02:25:46.9936814Z ok (5.757s)
2022-11-23T02:25:46.9937161Z   test_flatten_sharded_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-11-23T02:25:46.9937855Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63094
2022-11-23T02:25:46.9938405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63095
2022-11-23T02:25:46.9939069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9939510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9940232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9940716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9941286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9941749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9942336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9942813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9943260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:46.9943766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:46.9944432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9945148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9945664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:46.9946138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:46.9947119Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:46.9947700Z   warnings.warn(
2022-11-23T02:25:46.9948454Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:46.9949002Z   warnings.warn(
2022-11-23T02:25:46.9949262Z dist init r=0, world=2
2022-11-23T02:25:46.9949517Z dist init r=1, world=2
2022-11-23T02:25:46.9949747Z ok (4.813s)
2022-11-23T02:25:46.9950071Z   test_full_optim_state_dict_keys (__main__.TestFSDPOptimState)
2022-11-23T02:25:46.9950570Z Tests that the parameter keys returned by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63177
2022-11-23T02:25:46.9951077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63178
2022-11-23T02:25:46.9951702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9952166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9952756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9953225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9953822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9954276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9954836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9955319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9955783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:46.9956292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:46.9956939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9957652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9958256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:46.9958833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:46.9959183Z dist init r=0, world=2
2022-11-23T02:25:46.9959441Z dist init r=1, world=2
2022-11-23T02:25:46.9959679Z ok (3.811s)
2022-11-23T02:25:46.9960000Z   test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState)
2022-11-23T02:25:46.9960532Z Tests that :meth:`full_optim_state_dict` raises an error when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63260
2022-11-23T02:25:46.9961063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63261
2022-11-23T02:25:46.9961682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9962123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9962702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9963178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9963738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9964186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9964821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9965308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9965748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:46.9966247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:46.9966921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9967620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9968131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:46.9968608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:46.9968965Z dist init r=1, world=2
2022-11-23T02:25:46.9969201Z dist init r=0, world=2
2022-11-23T02:25:46.9969440Z ok (3.911s)
2022-11-23T02:25:46.9969748Z   test_optim_input_warning (__main__.TestFSDPOptimState)
2022-11-23T02:25:46.9970250Z Tests that passing the ``optim_input`` argument into optimizer state ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63343
2022-11-23T02:25:46.9970790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63344
2022-11-23T02:25:46.9971405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9972181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9972751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9973229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9973820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9974269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9974825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9975872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9976338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:46.9976826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:46.9977492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9978189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9978719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:46.9979173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:46.9979531Z dist init r=0, world=2
2022-11-23T02:25:46.9979785Z dist init r=1, world=2
2022-11-23T02:25:46.9980010Z ok (3.810s)
2022-11-23T02:25:46.9980508Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:46.9981188Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63426
2022-11-23T02:25:46.9981726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63427
2022-11-23T02:25:46.9982409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9982881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9983469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9983944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9984514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9984965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9985538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9986004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9986445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:46.9986950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:46.9987611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9988289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:46.9988818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:46.9989295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:46.9990209Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:46.9990762Z   warnings.warn(
2022-11-23T02:25:46.9991571Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:46.9992125Z   warnings.warn(
2022-11-23T02:25:46.9992374Z dist init r=1, world=2
2022-11-23T02:25:46.9992611Z dist init r=0, world=2
2022-11-23T02:25:46.9992921Z ok (3.911s)
2022-11-23T02:25:46.9993409Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:46.9994065Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63509
2022-11-23T02:25:46.9994610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63510
2022-11-23T02:25:46.9995233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9995690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9996252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9996730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9997317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:46.9997768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:46.9998327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:46.9998795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:46.9999254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:46.9999793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0000471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0001167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0001699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0002156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0003062Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0003622Z   warnings.warn(
2022-11-23T02:25:47.0004417Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0004943Z   warnings.warn(
2022-11-23T02:25:47.0005197Z dist init r=0, world=2
2022-11-23T02:25:47.0005456Z dist init r=1, world=2
2022-11-23T02:25:47.0005678Z ok (4.012s)
2022-11-23T02:25:47.0006164Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0006833Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63592
2022-11-23T02:25:47.0007372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63593
2022-11-23T02:25:47.0007978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0008429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0009014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0009489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0010135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0010588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0011166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0012024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0012490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0012993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0013661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0014338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0014875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0015353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0016270Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0016902Z   warnings.warn(
2022-11-23T02:25:47.0017721Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0018283Z   warnings.warn(
2022-11-23T02:25:47.0018535Z dist init r=1, world=2
2022-11-23T02:25:47.0018776Z dist init r=0, world=2
2022-11-23T02:25:47.0019017Z ok (4.011s)
2022-11-23T02:25:47.0019505Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0020161Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63675
2022-11-23T02:25:47.0020702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63676
2022-11-23T02:25:47.0021323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0021778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0022344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0022824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0023412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0023844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0024418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0024890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0025351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0025836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0026500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0027193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0027810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0028270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0029178Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0029744Z   warnings.warn(
2022-11-23T02:25:47.0030546Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0031078Z   warnings.warn(
2022-11-23T02:25:47.0031336Z dist init r=1, world=2
2022-11-23T02:25:47.0031588Z dist init r=0, world=2
2022-11-23T02:25:47.0031809Z ok (3.911s)
2022-11-23T02:25:47.0032300Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0032970Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63758
2022-11-23T02:25:47.0033582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63759
2022-11-23T02:25:47.0034198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0034654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0035237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0035718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0036286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0036731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0037310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0037758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0038220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0038719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0039381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0040063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0040589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0041064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0041981Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0042524Z   warnings.warn(
2022-11-23T02:25:47.0043324Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0043951Z   warnings.warn(
2022-11-23T02:25:47.0044201Z dist init r=1, world=2
2022-11-23T02:25:47.0044436Z dist init r=0, world=2
2022-11-23T02:25:47.0044677Z ok (3.911s)
2022-11-23T02:25:47.0045165Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0045826Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63841
2022-11-23T02:25:47.0046367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63842
2022-11-23T02:25:47.0046993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0047450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0048015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0048493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0049076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0049507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0050083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0050606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0051077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0051895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0052580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0053283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0053813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0054268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0055174Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0055743Z   warnings.warn(
2022-11-23T02:25:47.0056544Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0057081Z   warnings.warn(
2022-11-23T02:25:47.0057331Z dist init r=1, world=2
2022-11-23T02:25:47.0057585Z dist init r=0, world=2
2022-11-23T02:25:47.0057806Z ok (4.011s)
2022-11-23T02:25:47.0058292Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0059016Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63924
2022-11-23T02:25:47.0059567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63925
2022-11-23T02:25:47.0060167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0060621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0061203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0061758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0062349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0062797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0063373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0063827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0064287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0064788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0065449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0066134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0066661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0067138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0068118Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0068680Z   warnings.warn(
2022-11-23T02:25:47.0069487Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0070049Z   warnings.warn(
2022-11-23T02:25:47.0070282Z dist init r=1, world=2
2022-11-23T02:25:47.0070537Z dist init r=0, world=2
2022-11-23T02:25:47.0070777Z ok (3.912s)
2022-11-23T02:25:47.0071262Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0071921Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64007
2022-11-23T02:25:47.0072462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64008
2022-11-23T02:25:47.0073079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0073532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0074100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0074573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0075153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0075580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0076158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0076628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0077092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0077577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0078241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0079011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0079541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0080001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0080908Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0081471Z   warnings.warn(
2022-11-23T02:25:47.0082254Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0082805Z   warnings.warn(
2022-11-23T02:25:47.0083054Z dist init r=0, world=2
2022-11-23T02:25:47.0083306Z dist init r=1, world=2
2022-11-23T02:25:47.0083527Z ok (4.013s)
2022-11-23T02:25:47.0084017Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0084747Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64090
2022-11-23T02:25:47.0085302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64091
2022-11-23T02:25:47.0085903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0086353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0086941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0087399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0087983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0088430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0089012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0089464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0089921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0090422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0091069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0092180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0092712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0093188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0094049Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0094606Z   warnings.warn(
2022-11-23T02:25:47.0095367Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0096007Z   warnings.warn(
2022-11-23T02:25:47.0096794Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0097358Z   warnings.warn(
2022-11-23T02:25:47.0098162Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0098713Z   warnings.warn(
2022-11-23T02:25:47.0098943Z dist init r=0, world=2
2022-11-23T02:25:47.0099198Z dist init r=1, world=2
2022-11-23T02:25:47.0099439Z ok (4.011s)
2022-11-23T02:25:47.0099910Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0100588Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64173
2022-11-23T02:25:47.0101127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64174
2022-11-23T02:25:47.0101748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0102256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0102853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0103331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0103917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0104352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0104930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0105405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0105865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0106354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0107018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0107713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0108223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0108704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0109580Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0110132Z   warnings.warn(
2022-11-23T02:25:47.0110877Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0111425Z   warnings.warn(
2022-11-23T02:25:47.0112223Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0112851Z   warnings.warn(
2022-11-23T02:25:47.0113631Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0114182Z   warnings.warn(
2022-11-23T02:25:47.0114431Z dist init r=0, world=2
2022-11-23T02:25:47.0114686Z dist init r=1, world=2
2022-11-23T02:25:47.0114909Z ok (3.910s)
2022-11-23T02:25:47.0115403Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0116085Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64256
2022-11-23T02:25:47.0116610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64257
2022-11-23T02:25:47.0117230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0117680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0118261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0118718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0119371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0119833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0120418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0120870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0121334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0121833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0122481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0123177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0123707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0124185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0124527Z dist init r=1, world=2
2022-11-23T02:25:47.0124781Z dist init r=0, world=2
2022-11-23T02:25:47.0125026Z ok (3.309s)
2022-11-23T02:25:47.0125499Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0126177Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64335
2022-11-23T02:25:47.0126715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64336
2022-11-23T02:25:47.0127328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0127769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0128351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0128822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0129403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0129903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0130482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0130951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0131391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0132225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0132892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0133584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0134098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0134575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0134937Z dist init r=1, world=2
2022-11-23T02:25:47.0135172Z dist init r=0, world=2
2022-11-23T02:25:47.0135414Z ok (3.209s)
2022-11-23T02:25:47.0135906Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0136664Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64414
2022-11-23T02:25:47.0137202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64415
2022-11-23T02:25:47.0137820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0138279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0138863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0139320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0139901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0140349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0140910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0141379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0141835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0142332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0142980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0143672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0144204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0144682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0145543Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0146098Z   warnings.warn(
2022-11-23T02:25:47.0146857Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0147488Z   warnings.warn(
2022-11-23T02:25:47.0148278Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0148839Z   warnings.warn(
2022-11-23T02:25:47.0149642Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0150191Z   warnings.warn(
2022-11-23T02:25:47.0150423Z dist init r=0, world=2
2022-11-23T02:25:47.0150676Z dist init r=1, world=2
2022-11-23T02:25:47.0150916Z ok (3.910s)
2022-11-23T02:25:47.0151392Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0152067Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64497
2022-11-23T02:25:47.0152609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64498
2022-11-23T02:25:47.0153298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0153745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0154331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0154808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0155379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0155830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0156410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0156880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0157325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0157833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0158496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0159245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0159758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0160240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0161119Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0161672Z   warnings.warn(
2022-11-23T02:25:47.0162423Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0162963Z   warnings.warn(
2022-11-23T02:25:47.0163762Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0164393Z   warnings.warn(
2022-11-23T02:25:47.0165170Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0165733Z   warnings.warn(
2022-11-23T02:25:47.0165986Z dist init r=1, world=2
2022-11-23T02:25:47.0166225Z dist init r=0, world=2
2022-11-23T02:25:47.0166469Z ok (4.010s)
2022-11-23T02:25:47.0166958Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0167631Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64580
2022-11-23T02:25:47.0168159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64581
2022-11-23T02:25:47.0168772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0169226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0169788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0170326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0170927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0171378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0172312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0172793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0173256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0173764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0174411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0175106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0175635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0176095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0176457Z dist init r=0, world=2
2022-11-23T02:25:47.0176714Z dist init r=1, world=2
2022-11-23T02:25:47.0176963Z ok (3.310s)
2022-11-23T02:25:47.0177434Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0178105Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64659
2022-11-23T02:25:47.0178647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64660
2022-11-23T02:25:47.0179253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0179709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0180291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0180761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0181425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0181878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0182450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0182921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0183359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0183866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0184528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0185207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0185739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0186216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0186574Z dist init r=0, world=2
2022-11-23T02:25:47.0186814Z dist init r=1, world=2
2022-11-23T02:25:47.0187055Z ok (3.209s)
2022-11-23T02:25:47.0187494Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0188174Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64738
2022-11-23T02:25:47.0188715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64739
2022-11-23T02:25:47.0189336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0189799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0190363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0190839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0191425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0191874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0192436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0192903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0193359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0193841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0194508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0195199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0195724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0196182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0197091Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0197658Z   warnings.warn(
2022-11-23T02:25:47.0198459Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0199063Z   warnings.warn(
2022-11-23T02:25:47.0199314Z dist init r=0, world=2
2022-11-23T02:25:47.0199569Z dist init r=1, world=2
2022-11-23T02:25:47.0199791Z ok (4.011s)
2022-11-23T02:25:47.0200228Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0200846Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64821
2022-11-23T02:25:47.0201378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64822
2022-11-23T02:25:47.0201982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0202440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0203020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0203495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0204064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0204512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0205149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0205615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0206074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0206576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0207249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0207928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0208454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0208932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0209852Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0210401Z   warnings.warn(
2022-11-23T02:25:47.0211200Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0212209Z   warnings.warn(
2022-11-23T02:25:47.0212469Z dist init r=1, world=2
2022-11-23T02:25:47.0212706Z dist init r=0, world=2
2022-11-23T02:25:47.0212948Z ok (4.011s)
2022-11-23T02:25:47.0213394Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0213996Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64904
2022-11-23T02:25:47.0214529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64905
2022-11-23T02:25:47.0215147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0215700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0216268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0216746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0217334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0217763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0218404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0218882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0219345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0219830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0220498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0221195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0221722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0222180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0223121Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0223689Z   warnings.warn(
2022-11-23T02:25:47.0224438Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0224990Z   warnings.warn(
2022-11-23T02:25:47.0225789Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0226345Z   warnings.warn(
2022-11-23T02:25:47.0227149Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0227678Z   warnings.warn(
2022-11-23T02:25:47.0227928Z dist init r=1, world=2
2022-11-23T02:25:47.0228188Z dist init r=0, world=2
2022-11-23T02:25:47.0228411Z ok (4.010s)
2022-11-23T02:25:47.0228854Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0229471Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64987
2022-11-23T02:25:47.0230000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64988
2022-11-23T02:25:47.0230604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0231062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0231642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0232100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0232684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0233237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0233823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0234275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0234737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0235242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0235890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0236584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0237112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0237592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0238451Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0239000Z   warnings.warn(
2022-11-23T02:25:47.0239812Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0240363Z   warnings.warn(
2022-11-23T02:25:47.0241146Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0241709Z   warnings.warn(
2022-11-23T02:25:47.0242511Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0243067Z   warnings.warn(
2022-11-23T02:25:47.0243300Z dist init r=0, world=2
2022-11-23T02:25:47.0243555Z dist init r=1, world=2
2022-11-23T02:25:47.0243798Z ok (4.011s)
2022-11-23T02:25:47.0244107Z   test_rekey_optim_state_dict_to_names (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0244630Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65070
2022-11-23T02:25:47.0245160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65071
2022-11-23T02:25:47.0245778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0246219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0246799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0247277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0247842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0248294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0248873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0249341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0249783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0250352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0251016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0251958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0252470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0252955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0253871Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0254445Z   warnings.warn(
2022-11-23T02:25:47.0255231Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0255789Z   warnings.warn(
2022-11-23T02:25:47.0256042Z dist init r=1, world=2
2022-11-23T02:25:47.0256277Z dist init r=0, world=2
2022-11-23T02:25:47.0256516Z ok (4.010s)
2022-11-23T02:25:47.0256999Z   test_save_load_without_0th_param_state_state_dict_type_StateDictType_FULL_STATE_DICT (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0257612Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65153
2022-11-23T02:25:47.0258138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65154
2022-11-23T02:25:47.0258802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0259265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0259831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0260310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0260893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0261345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0261901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0262373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0262832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0263339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0263987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0264683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0265213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0265675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0266038Z dist init r=1, world=2
2022-11-23T02:25:47.0266294Z dist init r=0, world=2
2022-11-23T02:25:47.0266538Z ok (3.712s)
2022-11-23T02:25:47.0266929Z   test_save_load_without_0th_param_state_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0267527Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65236
2022-11-23T02:25:47.0268158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65237
2022-11-23T02:25:47.0268759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0269215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0269798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0270274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0270843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0271291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0271869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0272329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0272790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0273287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0273949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0274691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0275230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0275708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0276588Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0277116Z   warnings.warn(
2022-11-23T02:25:47.0277875Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0278419Z   warnings.warn(
2022-11-23T02:25:47.0278673Z dist init r=0, world=2
2022-11-23T02:25:47.0278911Z dist init r=1, world=2
2022-11-23T02:25:47.0279152Z ok (3.711s)
2022-11-23T02:25:47.0279511Z   test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0280187Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65319
2022-11-23T02:25:47.0280732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65320
2022-11-23T02:25:47.0281342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0281795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0282357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0282831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0283421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0283852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0284428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0284976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0285437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0285921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0286589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0287291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0287821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0288281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0288768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:47.0289266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:47.0289909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:47.0290600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:47.0291814Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0292405Z   warnings.warn(
2022-11-23T02:25:47.0293198Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0293759Z   warnings.warn(
2022-11-23T02:25:47.0294147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:25:47.0294654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:25:47.0295297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:47.0295993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:47.0296397Z dist init r=0, world=2
2022-11-23T02:25:47.0296652Z dist init r=1, world=2
2022-11-23T02:25:47.0296877Z ok (4.212s)
2022-11-23T02:25:47.0297319Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0298088Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65412
2022-11-23T02:25:47.0298616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65413
2022-11-23T02:25:47.0299232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0299689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0300268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0300731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0301315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0301762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0302320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0302882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0303342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0303845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0304492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0305195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0305724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0306199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0307097Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0307657Z   warnings.warn(
2022-11-23T02:25:47.0308464Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0309092Z   warnings.warn(
2022-11-23T02:25:47.0309338Z dist init r=0, world=2
2022-11-23T02:25:47.0309591Z dist init r=1, world=2
2022-11-23T02:25:47.0309826Z ok (4.212s)
2022-11-23T02:25:47.0310249Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0311019Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65495
2022-11-23T02:25:47.0311568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65496
2022-11-23T02:25:47.0312179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0312615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0313188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0313662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0314247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0314677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0315253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0315723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0316162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0316658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0317314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0318007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0318517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0318990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0319897Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0320536Z   warnings.warn(
2022-11-23T02:25:47.0321315Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0321868Z   warnings.warn(
2022-11-23T02:25:47.0322117Z dist init r=1, world=2
2022-11-23T02:25:47.0322350Z dist init r=0, world=2
2022-11-23T02:25:47.0322587Z ok (4.211s)
2022-11-23T02:25:47.0323023Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0323786Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65578
2022-11-23T02:25:47.0324315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65579
2022-11-23T02:25:47.0324928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0325378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0325956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0326468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0327063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0327512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0328073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0328546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0329002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0329498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0330144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0330839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0331364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0332023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0332925Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0333496Z   warnings.warn(
2022-11-23T02:25:47.0334293Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0334839Z   warnings.warn(
2022-11-23T02:25:47.0335073Z dist init r=1, world=2
2022-11-23T02:25:47.0335325Z dist init r=0, world=2
2022-11-23T02:25:47.0335564Z ok (4.111s)
2022-11-23T02:25:47.0335982Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0336738Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65661
2022-11-23T02:25:47.0337376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65662
2022-11-23T02:25:47.0337996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0338434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0339012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0339488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0340054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0340499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0341072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0341542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0341982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0342482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0343145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0343909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0344434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0344912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0345837Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0346405Z   warnings.warn(
2022-11-23T02:25:47.0347185Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0347737Z   warnings.warn(
2022-11-23T02:25:47.0347986Z dist init r=1, world=2
2022-11-23T02:25:47.0348220Z dist init r=0, world=2
2022-11-23T02:25:47.0348458Z ok (4.111s)
2022-11-23T02:25:47.0348894Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0349654Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65744
2022-11-23T02:25:47.0350182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65745
2022-11-23T02:25:47.0350793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0351250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0351811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0352288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0352866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0353309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0353868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0354407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0354862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0355344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0356008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0356709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0357233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0357690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0358594Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0359210Z   warnings.warn(
2022-11-23T02:25:47.0360014Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0360545Z   warnings.warn(
2022-11-23T02:25:47.0360849Z dist init r=0, world=2
2022-11-23T02:25:47.0361110Z dist init r=1, world=2
2022-11-23T02:25:47.0361331Z ok (4.312s)
2022-11-23T02:25:47.0361770Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0362533Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65827
2022-11-23T02:25:47.0363077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65828
2022-11-23T02:25:47.0363670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0364124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0364700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0365176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0365744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0366190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0366760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0367214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0367673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0368176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0368835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0369513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0370039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0370511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0371413Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0372222Z   warnings.warn(
2022-11-23T02:25:47.0373030Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0373593Z   warnings.warn(
2022-11-23T02:25:47.0373844Z dist init r=1, world=2
2022-11-23T02:25:47.0374078Z dist init r=0, world=2
2022-11-23T02:25:47.0374313Z ok (4.311s)
2022-11-23T02:25:47.0374751Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0375494Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65910
2022-11-23T02:25:47.0376038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65911
2022-11-23T02:25:47.0376646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0377096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0377659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0378202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0378808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0379240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0379811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0380287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0380742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0381222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0381883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0382581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0383105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0383560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0384457Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0385026Z   warnings.warn(
2022-11-23T02:25:47.0385825Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0386359Z   warnings.warn(
2022-11-23T02:25:47.0386609Z dist init r=1, world=2
2022-11-23T02:25:47.0386862Z dist init r=0, world=2
2022-11-23T02:25:47.0387082Z ok (4.112s)
2022-11-23T02:25:47.0387513Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0388270Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65993
2022-11-23T02:25:47.0388872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65994
2022-11-23T02:25:47.0389470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0389919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0390502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0390964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0391547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0391990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0392562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0393018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0393475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0393973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0394637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0395382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0395924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0396396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0397307Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0397854Z   warnings.warn(
2022-11-23T02:25:47.0398650Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0399201Z   warnings.warn(
2022-11-23T02:25:47.0399436Z dist init r=1, world=2
2022-11-23T02:25:47.0399689Z dist init r=0, world=2
2022-11-23T02:25:47.0399928Z ok (4.112s)
2022-11-23T02:25:47.0400271Z   test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0400925Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66076
2022-11-23T02:25:47.0401459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66077
2022-11-23T02:25:47.0402070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0402502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0403084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0403551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0404131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0404560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0405131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0405664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0406104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0406604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0407268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0407959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0408471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0408944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0409419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:47.0409915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:47.0410552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:47.0411235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:47.0411949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:25:47.0412527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:25:47.0413187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:47.0413878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:47.0414282Z dist init r=1, world=2
2022-11-23T02:25:47.0414522Z dist init r=0, world=2
2022-11-23T02:25:47.0414759Z ok (4.511s)
2022-11-23T02:25:47.0415114Z   test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0415812Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66169
2022-11-23T02:25:47.0416334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66170
2022-11-23T02:25:47.0416947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0417398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0417963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0418437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0419022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0419470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0420028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0420497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0420952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0421439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0422098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0422789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0423402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0423857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0424339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:47.0424833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:47.0425496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:47.0426170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:47.0427129Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0427692Z   warnings.warn(
2022-11-23T02:25:47.0428493Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0429020Z   warnings.warn(
2022-11-23T02:25:47.0429400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:25:47.0429950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:25:47.0430623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:47.0431292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:47.0431701Z dist init r=0, world=2
2022-11-23T02:25:47.0431953Z dist init r=1, world=2
2022-11-23T02:25:47.0432175Z ok (4.111s)
2022-11-23T02:25:47.0432609Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0433374Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66262
2022-11-23T02:25:47.0433917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66263
2022-11-23T02:25:47.0434514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0434968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0435542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0436000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0436580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0437023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0437594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0438045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0438507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0439006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0439663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0440339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0440936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0441406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0442317Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0442868Z   warnings.warn(
2022-11-23T02:25:47.0443671Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0444223Z   warnings.warn(
2022-11-23T02:25:47.0444457Z dist init r=0, world=2
2022-11-23T02:25:47.0444707Z dist init r=1, world=2
2022-11-23T02:25:47.0444941Z ok (4.211s)
2022-11-23T02:25:47.0445379Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0446126Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66345
2022-11-23T02:25:47.0446664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66346
2022-11-23T02:25:47.0447330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0447777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0448361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0448838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0449419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0449847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0450422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0450886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0451344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0452006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0452671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0453368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0453879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0454352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0455252Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0455815Z   warnings.warn(
2022-11-23T02:25:47.0456597Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0457143Z   warnings.warn(
2022-11-23T02:25:47.0457491Z dist init r=0, world=2
2022-11-23T02:25:47.0457746Z dist init r=1, world=2
2022-11-23T02:25:47.0457969Z ok (4.211s)
2022-11-23T02:25:47.0458407Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0459220Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66428
2022-11-23T02:25:47.0459753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66429
2022-11-23T02:25:47.0460372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0460823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0461395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0461853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0462436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0462879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0463451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0463903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0464433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0464949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0465596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0466294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0466824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0467298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0468189Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0468755Z   warnings.warn(
2022-11-23T02:25:47.0469411Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0469523Z   warnings.warn(
2022-11-23T02:25:47.0469639Z dist init r=0, world=2
2022-11-23T02:25:47.0469749Z dist init r=1, world=2
2022-11-23T02:25:47.0469850Z ok (4.111s)
2022-11-23T02:25:47.0470143Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0470595Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66511
2022-11-23T02:25:47.0470817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66512
2022-11-23T02:25:47.0471193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0471369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0471752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0471943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0472388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0472555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0491675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0491948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0492216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0492469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0492921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0493311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0493553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0493788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0494461Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0494715Z   warnings.warn(
2022-11-23T02:25:47.0495407Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0495518Z   warnings.warn(
2022-11-23T02:25:47.0495635Z dist init r=0, world=2
2022-11-23T02:25:47.0495744Z dist init r=1, world=2
2022-11-23T02:25:47.0495827Z ok (4.112s)
2022-11-23T02:25:47.0496138Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0496599Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66594
2022-11-23T02:25:47.0496823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66595
2022-11-23T02:25:47.0497204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0497384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0497771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0497964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0498341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0498501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0498887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0499081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0499336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0499585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0499991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0500391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0500747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0500963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0501632Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0501753Z   warnings.warn(
2022-11-23T02:25:47.0502415Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0502526Z   warnings.warn(
2022-11-23T02:25:47.0502642Z dist init r=1, world=2
2022-11-23T02:25:47.0502751Z dist init r=0, world=2
2022-11-23T02:25:47.0502852Z ok (4.213s)
2022-11-23T02:25:47.0503161Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0503599Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66677
2022-11-23T02:25:47.0503828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66678
2022-11-23T02:25:47.0504254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0504440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0504828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0505023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0505394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0505570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0505948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0506126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0506378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0506626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0507031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0507433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0507670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0507902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0508564Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0508681Z   warnings.warn(
2022-11-23T02:25:47.0509325Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0509440Z   warnings.warn(
2022-11-23T02:25:47.0509552Z dist init r=0, world=2
2022-11-23T02:25:47.0509739Z dist init r=1, world=2
2022-11-23T02:25:47.0509843Z ok (4.211s)
2022-11-23T02:25:47.0510150Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0510604Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66760
2022-11-23T02:25:47.0510829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66761
2022-11-23T02:25:47.0511205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0511365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0511746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0511939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0512308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0512482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0512862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0513051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0513352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0513589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0513998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0514403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0514639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0514870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0515532Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0515651Z   warnings.warn(
2022-11-23T02:25:47.0516311Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0516423Z   warnings.warn(
2022-11-23T02:25:47.0516535Z dist init r=1, world=2
2022-11-23T02:25:47.0516632Z dist init r=0, world=2
2022-11-23T02:25:47.0516735Z ok (4.011s)
2022-11-23T02:25:47.0517044Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0517499Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66843
2022-11-23T02:25:47.0517725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66844
2022-11-23T02:25:47.0518101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0518340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0518729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0518907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0519345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0519520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0519895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0520088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0520340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0520587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0520990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0521389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0521608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0521839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0522501Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0522664Z   warnings.warn(
2022-11-23T02:25:47.0523335Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0523446Z   warnings.warn(
2022-11-23T02:25:47.0523557Z dist init r=1, world=2
2022-11-23T02:25:47.0523671Z dist init r=0, world=2
2022-11-23T02:25:47.0523754Z ok (4.111s)
2022-11-23T02:25:47.0523967Z   test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0524399Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66926
2022-11-23T02:25:47.0524623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66927
2022-11-23T02:25:47.0524995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0525176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0525560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0525753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0526122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0526283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0526662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0526856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0527104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0527356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0527760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0528162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0528453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0528686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0528915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:47.0529160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:47.0529560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:47.0529962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:47.0530206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:25:47.0530447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:25:47.0530842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:47.0531239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:47.0531353Z dist init r=1, world=2
2022-11-23T02:25:47.0531612Z dist init r=0, world=2
2022-11-23T02:25:47.0531722Z ok (4.511s)
2022-11-23T02:25:47.0532130Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0532460Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67019
2022-11-23T02:25:47.0532682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67020
2022-11-23T02:25:47.0533069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0533250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0533634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0533811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0534178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0534355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0534734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0534923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0535171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0535419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0535821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0536219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0536435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0536662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0537328Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0537443Z   warnings.warn(
2022-11-23T02:25:47.0538176Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0538286Z   warnings.warn(
2022-11-23T02:25:47.0538399Z dist init r=0, world=2
2022-11-23T02:25:47.0538508Z dist init r=1, world=2
2022-11-23T02:25:47.0538591Z ok (3.910s)
2022-11-23T02:25:47.0538919Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0539238Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67102
2022-11-23T02:25:47.0539461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67103
2022-11-23T02:25:47.0539835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0540016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0540400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0540594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0540967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0541184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0541576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0541769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0542020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0542272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0542675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0543074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0543309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0543543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0544187Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0544302Z   warnings.warn(
2022-11-23T02:25:47.0544963Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0545073Z   warnings.warn(
2022-11-23T02:25:47.0545186Z dist init r=0, world=2
2022-11-23T02:25:47.0545296Z dist init r=1, world=2
2022-11-23T02:25:47.0545396Z ok (3.910s)
2022-11-23T02:25:47.0545728Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0546043Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67185
2022-11-23T02:25:47.0546250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67186
2022-11-23T02:25:47.0546624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0546866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0547251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0547448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0547812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0547989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0548363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0548558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0548787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0549038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0549440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0549836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0550069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0550351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0550992Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0551107Z   warnings.warn(
2022-11-23T02:25:47.0551723Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0551817Z   warnings.warn(
2022-11-23T02:25:47.0552477Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0552586Z   warnings.warn(
2022-11-23T02:25:47.0553249Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0553359Z   warnings.warn(
2022-11-23T02:25:47.0553472Z dist init r=0, world=2
2022-11-23T02:25:47.0553589Z dist init r=1, world=2
2022-11-23T02:25:47.0553827Z ok (3.910s)
2022-11-23T02:25:47.0554295Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0554941Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67268
2022-11-23T02:25:47.0555486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67269
2022-11-23T02:25:47.0556098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0556558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0557138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0557619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0558269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0558772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0559361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0559833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0560279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0560786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0561457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0562142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0562678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0563154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0564025Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0564561Z   warnings.warn(
2022-11-23T02:25:47.0565418Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:25:47.0565985Z   warnings.warn(
2022-11-23T02:25:47.0566787Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0567358Z   warnings.warn(
2022-11-23T02:25:47.0568135Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2022-11-23T02:25:47.0568709Z   warnings.warn(
2022-11-23T02:25:47.0568962Z dist init r=1, world=2
2022-11-23T02:25:47.0569243Z dist init r=0, world=2
2022-11-23T02:25:47.0569467Z ok (4.011s)
2022-11-23T02:25:47.0569777Z   test_use_orig_params_error (__main__.TestFSDPOptimState)
2022-11-23T02:25:47.0570296Z Tests that the optimizer state checkpointing APIs raise an error ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67351
2022-11-23T02:25:47.0570818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67352
2022-11-23T02:25:47.0571671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0572141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0572731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0573188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0573775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:47.0574223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:47.0574777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:47.0575246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:47.0575809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:47.0576317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:47.0576966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0577662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:47.0578194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:47.0578678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:47.0579042Z dist init r=0, world=2
2022-11-23T02:25:47.0579302Z dist init r=1, world=2
2022-11-23T02:25:47.0579545Z ok (3.810s)
2022-11-23T02:25:47.0579678Z 
2022-11-23T02:25:47.0579956Z ----------------------------------------------------------------------
2022-11-23T02:25:47.0580298Z Ran 53 tests in 213.733s
2022-11-23T02:25:47.0580464Z 
2022-11-23T02:25:47.0580568Z OK
2022-11-23T02:25:47.0580712Z 
2022-11-23T02:25:47.0580822Z Generating XML reports...
2022-11-23T02:25:47.0581457Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221123022212.xml
2022-11-23T02:25:47.0581835Z 
2022-11-23T02:25:47.0582259Z ##[endgroup]
2022-11-23T02:25:47.0582972Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_q6gsm50a)
2022-11-23T02:25:47.0583351Z 
2022-11-23T02:25:47.0583634Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-11-23 02:25:46.992330]
2022-11-23T02:25:47.0584311Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:25:46.992596]
2022-11-23T02:32:25.7724938Z 
2022-11-23T02:32:25.7727749Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict
2022-11-23T02:32:25.7729161Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_rtijl0w0)
2022-11-23T02:32:25.7729880Z 
2022-11-23T02:32:25.7730103Z Running tests...
2022-11-23T02:32:25.7734474Z ----------------------------------------------------------------------
2022-11-23T02:32:25.7735128Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict
2022-11-23T02:32:25.7735815Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7736466Z Tests that we can save a state_dict and load it into a blank model ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:25.7736969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67469
2022-11-23T02:32:25.7737410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67470
2022-11-23T02:32:25.7738065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7738534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7739105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7739593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7740191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7740642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7741203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7741989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7742458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7742973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7745555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7746888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7747954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7748888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7749564Z dist init r=1, world=2
2022-11-23T02:32:25.7750010Z dist init r=0, world=2
2022-11-23T02:32:25.7750443Z ok (4.966s)
2022-11-23T02:32:25.7751358Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7752585Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67548
2022-11-23T02:32:25.7753820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67549
2022-11-23T02:32:25.7755028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7755905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7756965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7757685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7758298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7758759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7759324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7760060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7760768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7761258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7761954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7762970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7763510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7763979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7764344Z dist init r=0, world=2
2022-11-23T02:32:25.7764604Z dist init r=1, world=2
2022-11-23T02:32:25.7764849Z ok (3.310s)
2022-11-23T02:32:25.7765366Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7766069Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67627
2022-11-23T02:32:25.7766605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67628
2022-11-23T02:32:25.7767374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7767824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7768413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7768887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7769461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7769924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7770508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7770983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7771944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7772559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7773246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7773949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7774568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7775067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7775435Z dist init r=0, world=2
2022-11-23T02:32:25.7775679Z dist init r=1, world=2
2022-11-23T02:32:25.7775923Z ok (3.311s)
2022-11-23T02:32:25.7776449Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7777159Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67706
2022-11-23T02:32:25.7777677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67707
2022-11-23T02:32:25.7778302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7778767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7779357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7779894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7780482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7780951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7781511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7781994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7782460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7782972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7783622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7784326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7784854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7785427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7785775Z dist init r=1, world=2
2022-11-23T02:32:25.7786039Z dist init r=0, world=2
2022-11-23T02:32:25.7786292Z ok (3.210s)
2022-11-23T02:32:25.7786796Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7787502Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67785
2022-11-23T02:32:25.7788039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67786
2022-11-23T02:32:25.7788674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7789121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7789705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7790187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7790773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7791207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7791837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7792319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7792781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7793265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7793929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7794619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7795146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7795603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7795952Z dist init r=1, world=2
2022-11-23T02:32:25.7796211Z dist init r=0, world=2
2022-11-23T02:32:25.7796439Z ok (3.411s)
2022-11-23T02:32:25.7796954Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7797640Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67864
2022-11-23T02:32:25.7798175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67865
2022-11-23T02:32:25.7798773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7799228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7799805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7800264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7800844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7801295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7801866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7802374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7802834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7803334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7803993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7804667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7805195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7805671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7806014Z dist init r=1, world=2
2022-11-23T02:32:25.7806275Z dist init r=0, world=2
2022-11-23T02:32:25.7806518Z ok (3.210s)
2022-11-23T02:32:25.7807017Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7807707Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67943
2022-11-23T02:32:25.7808239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67944
2022-11-23T02:32:25.7808902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7809346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7809928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7810403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7810982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7811412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7812523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7812992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7813454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7813937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7814597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7815297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7815809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7816286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7816643Z dist init r=1, world=2
2022-11-23T02:32:25.7816901Z dist init r=0, world=2
2022-11-23T02:32:25.7817127Z ok (3.211s)
2022-11-23T02:32:25.7817644Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7818334Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68022
2022-11-23T02:32:25.7818848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68023
2022-11-23T02:32:25.7819599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7820054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7820628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7821083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7821671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7822122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7822698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7823149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7823611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7824116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7824762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7825456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7826050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7826538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7826883Z dist init r=0, world=2
2022-11-23T02:32:25.7827141Z dist init r=1, world=2
2022-11-23T02:32:25.7827385Z ok (3.310s)
2022-11-23T02:32:25.7827886Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7828587Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68101
2022-11-23T02:32:25.7829121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68102
2022-11-23T02:32:25.7829739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7830179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7830757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7831229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7831809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7832241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7832810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7833275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7833715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7834215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7834878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7835570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7836075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7836621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7836985Z dist init r=0, world=2
2022-11-23T02:32:25.7837223Z dist init r=1, world=2
2022-11-23T02:32:25.7837469Z ok (3.411s)
2022-11-23T02:32:25.7837990Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7838685Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68180
2022-11-23T02:32:25.7839199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68181
2022-11-23T02:32:25.7839815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7840269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7840873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7841332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7841928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7842373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7842984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7843462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7843919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7844417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7845065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7845757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7846281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7846756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7847095Z dist init r=0, world=2
2022-11-23T02:32:25.7847357Z dist init r=1, world=2
2022-11-23T02:32:25.7847601Z ok (3.210s)
2022-11-23T02:32:25.7848100Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7848788Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68259
2022-11-23T02:32:25.7849323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68260
2022-11-23T02:32:25.7849940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7850377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7850981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7851826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7852515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7852963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7853539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7854107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7854548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7855053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7855717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7856410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7856917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7857395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7857758Z dist init r=1, world=2
2022-11-23T02:32:25.7858000Z dist init r=0, world=2
2022-11-23T02:32:25.7858245Z ok (3.210s)
2022-11-23T02:32:25.7858761Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7859450Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68338
2022-11-23T02:32:25.7859966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68339
2022-11-23T02:32:25.7860662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7861128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7861712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7862174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7862757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7863204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7863760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7864230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7864690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7865191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7865834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7866528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7867060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7867538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7867879Z dist init r=1, world=2
2022-11-23T02:32:25.7868134Z dist init r=0, world=2
2022-11-23T02:32:25.7868376Z ok (3.210s)
2022-11-23T02:32:25.7868877Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7869566Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68417
2022-11-23T02:32:25.7870094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68418
2022-11-23T02:32:25.7870771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7871207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7871785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7872260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7872826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7873274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7873848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7874317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7874756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7875258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7875921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7876613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7877169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7877653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7878012Z dist init r=1, world=2
2022-11-23T02:32:25.7878248Z dist init r=0, world=2
2022-11-23T02:32:25.7878492Z ok (3.411s)
2022-11-23T02:32:25.7879006Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7879752Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68496
2022-11-23T02:32:25.7880272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68497
2022-11-23T02:32:25.7880893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7881353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7881914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7882390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7882971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7883424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7883979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7884448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7884904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7885384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7886051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7886742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7887270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7887790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7888149Z dist init r=1, world=2
2022-11-23T02:32:25.7888406Z dist init r=0, world=2
2022-11-23T02:32:25.7888632Z ok (3.210s)
2022-11-23T02:32:25.7889145Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7889838Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68575
2022-11-23T02:32:25.7890375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68576
2022-11-23T02:32:25.7890976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7891787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7892509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7892983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7893551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7894006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7894668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7895149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7895592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7896095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7896758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7897432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7897958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7898438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7898798Z dist init r=1, world=2
2022-11-23T02:32:25.7899039Z dist init r=0, world=2
2022-11-23T02:32:25.7899285Z ok (3.310s)
2022-11-23T02:32:25.7899801Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7900469Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68654
2022-11-23T02:32:25.7901011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68655
2022-11-23T02:32:25.7901626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7902080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7902641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7903121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7903703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7904153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7904707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7905264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7905723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7906207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7906873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7907572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7908102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7908560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7908915Z dist init r=0, world=2
2022-11-23T02:32:25.7909180Z dist init r=1, world=2
2022-11-23T02:32:25.7909407Z ok (3.210s)
2022-11-23T02:32:25.7909934Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7910632Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68733
2022-11-23T02:32:25.7911164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68734
2022-11-23T02:32:25.7911814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7912278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7912861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7913342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7913902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7914354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7914929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7915379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7915841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7916339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7917002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7917672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7918203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7918677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7919042Z dist init r=0, world=2
2022-11-23T02:32:25.7919279Z dist init r=1, world=2
2022-11-23T02:32:25.7919523Z ok (3.311s)
2022-11-23T02:32:25.7920054Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7920732Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68812
2022-11-23T02:32:25.7921268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68813
2022-11-23T02:32:25.7921950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7922409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7922967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7923443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7924034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7924463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7925037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7925506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7925964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7926450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7927113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7927805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7928383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7928850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7929203Z dist init r=0, world=2
2022-11-23T02:32:25.7929459Z dist init r=1, world=2
2022-11-23T02:32:25.7929683Z ok (3.310s)
2022-11-23T02:32:25.7930210Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7930911Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68891
2022-11-23T02:32:25.7931795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68892
2022-11-23T02:32:25.7932631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7933092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7933675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7934152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7934711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7935166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7935741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7936193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7936654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7937152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7937814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7938491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7939019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7939595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7939941Z dist init r=0, world=2
2022-11-23T02:32:25.7940200Z dist init r=1, world=2
2022-11-23T02:32:25.7940447Z ok (3.210s)
2022-11-23T02:32:25.7940969Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7941654Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68970
2022-11-23T02:32:25.7942189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68971
2022-11-23T02:32:25.7942807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7943266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7943830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7944306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7944888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7945316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7945961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7946440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7946901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7947383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7948049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7948740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7949265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7949719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7950073Z dist init r=1, world=2
2022-11-23T02:32:25.7950334Z dist init r=0, world=2
2022-11-23T02:32:25.7950559Z ok (3.311s)
2022-11-23T02:32:25.7951078Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7951774Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69049
2022-11-23T02:32:25.7952308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69050
2022-11-23T02:32:25.7952909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7953361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7953938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7954397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7954981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7955425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7955997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7956509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7956968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7957470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7958116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7958813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7959341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7959821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7960163Z dist init r=1, world=2
2022-11-23T02:32:25.7960423Z dist init r=0, world=2
2022-11-23T02:32:25.7960667Z ok (3.411s)
2022-11-23T02:32:25.7961170Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7961866Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69128
2022-11-23T02:32:25.7962397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69129
2022-11-23T02:32:25.7963059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7963504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7964086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7964562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7965143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7965569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7966143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7966609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7967073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7967556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7968214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7968902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7969412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7969887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7970240Z dist init r=0, world=2
2022-11-23T02:32:25.7970495Z dist init r=1, world=2
2022-11-23T02:32:25.7970717Z ok (3.411s)
2022-11-23T02:32:25.7971239Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7972442Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69207
2022-11-23T02:32:25.7972962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69208
2022-11-23T02:32:25.7973687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7974142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7974717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7975175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7975763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7976210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7976788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7977241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7977701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7978208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7978851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7979547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7980210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7980698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7981041Z dist init r=0, world=2
2022-11-23T02:32:25.7981299Z dist init r=1, world=2
2022-11-23T02:32:25.7981545Z ok (3.210s)
2022-11-23T02:32:25.7982044Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7982744Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69286
2022-11-23T02:32:25.7983276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69287
2022-11-23T02:32:25.7983893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7984333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7984913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7985387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7985971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7986404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7986973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7987442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7987884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7988384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7989048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7989739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7990248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.7990790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.7991152Z dist init r=0, world=2
2022-11-23T02:32:25.7991390Z dist init r=1, world=2
2022-11-23T02:32:25.7991635Z ok (3.210s)
2022-11-23T02:32:25.7992158Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.7992851Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69365
2022-11-23T02:32:25.7993366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69366
2022-11-23T02:32:25.7993984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7994439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7995019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7995476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7996057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.7996505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.7997113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.7997594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.7998052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.7998551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.7999197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.7999886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8000410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8000887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8001226Z dist init r=1, world=2
2022-11-23T02:32:25.8001485Z dist init r=0, world=2
2022-11-23T02:32:25.8001730Z ok (3.411s)
2022-11-23T02:32:25.8002235Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8002931Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69444
2022-11-23T02:32:25.8003464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69445
2022-11-23T02:32:25.8004076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8004511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8005088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8005562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8006126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8006577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8007149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8007680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8008120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8008619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8009281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8009981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8010489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8010965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8011325Z dist init r=1, world=2
2022-11-23T02:32:25.8012006Z dist init r=0, world=2
2022-11-23T02:32:25.8012261Z ok (3.311s)
2022-11-23T02:32:25.8012789Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8013487Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69523
2022-11-23T02:32:25.8014003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69524
2022-11-23T02:32:25.8014709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8015176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8015759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8016216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8016797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8017244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8017800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8018269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8018733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8019235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8019883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8020572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8021104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8021581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8021923Z dist init r=1, world=2
2022-11-23T02:32:25.8022180Z dist init r=0, world=2
2022-11-23T02:32:25.8022424Z ok (3.211s)
2022-11-23T02:32:25.8022926Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8023615Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69602
2022-11-23T02:32:25.8024146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69603
2022-11-23T02:32:25.8024858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8025292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8025866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8026339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8026905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8027354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8027927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8028395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8028834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8029339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8029999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8030699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8031263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8031748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8032109Z dist init r=0, world=2
2022-11-23T02:32:25.8032346Z dist init r=1, world=2
2022-11-23T02:32:25.8032590Z ok (3.311s)
2022-11-23T02:32:25.8033116Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8033817Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69681
2022-11-23T02:32:25.8034330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69682
2022-11-23T02:32:25.8034946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8035402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8035964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8036440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8037028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8037478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8038036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8038510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8038967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8039446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8040111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8040803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8041326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8041841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8042202Z dist init r=1, world=2
2022-11-23T02:32:25.8042457Z dist init r=0, world=2
2022-11-23T02:32:25.8042685Z ok (3.411s)
2022-11-23T02:32:25.8043203Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8043961Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69760
2022-11-23T02:32:25.8044496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69761
2022-11-23T02:32:25.8045100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8045557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8046136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8046613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8047179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8047629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8048253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8048731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8049172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8049670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8050335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8051006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8051918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8052498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8052863Z dist init r=1, world=2
2022-11-23T02:32:25.8053109Z dist init r=0, world=2
2022-11-23T02:32:25.8053359Z ok (3.311s)
2022-11-23T02:32:25.8053880Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8054555Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69839
2022-11-23T02:32:25.8055091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69840
2022-11-23T02:32:25.8055722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8056180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8056743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8057221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8057809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8058261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8058816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8059392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8059853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8060336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8060999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8061692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8062223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8062683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8063039Z dist init r=0, world=2
2022-11-23T02:32:25.8063298Z dist init r=1, world=2
2022-11-23T02:32:25.8063526Z ok (3.311s)
2022-11-23T02:32:25.8064045Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8064740Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69918
2022-11-23T02:32:25.8065275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69919
2022-11-23T02:32:25.8065935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8066406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8066993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8067470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8068032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8068486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8069056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8069504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8069959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8070459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8071115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8071785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8072315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8072788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8073150Z dist init r=1, world=2
2022-11-23T02:32:25.8073387Z dist init r=0, world=2
2022-11-23T02:32:25.8073631Z ok (3.311s)
2022-11-23T02:32:25.8074152Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8074819Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69997
2022-11-23T02:32:25.8075349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69998
2022-11-23T02:32:25.8076030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8076485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8077043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8077517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8078103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8078534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8079108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8079579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8080091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8080581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8081243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8081937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8082546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8083013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8083364Z dist init r=0, world=2
2022-11-23T02:32:25.8083619Z dist init r=1, world=2
2022-11-23T02:32:25.8083844Z ok (3.310s)
2022-11-23T02:32:25.8084355Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8085046Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70076
2022-11-23T02:32:25.8085580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70077
2022-11-23T02:32:25.8086182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8086643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8087220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8087677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8088258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8088713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8089287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8089737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8090194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8090695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8091354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8092565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8093090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8093663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8094009Z dist init r=0, world=2
2022-11-23T02:32:25.8094266Z dist init r=1, world=2
2022-11-23T02:32:25.8094511Z ok (3.310s)
2022-11-23T02:32:25.8095024Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8095693Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70155
2022-11-23T02:32:25.8096226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70156
2022-11-23T02:32:25.8096845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8097297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8097864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8098332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8098907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8099334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8099993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8100472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8100929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8101411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8102079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8102769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8103295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8103753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8104108Z dist init r=1, world=2
2022-11-23T02:32:25.8104353Z dist init r=0, world=2
2022-11-23T02:32:25.8104600Z ok (3.411s)
2022-11-23T02:32:25.8105107Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8105771Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70234
2022-11-23T02:32:25.8106306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70235
2022-11-23T02:32:25.8106917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8107371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8107927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8108400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8108982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8109427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8109982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8110508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8110961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8111447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8112102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8112794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8113316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8113769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8114116Z dist init r=1, world=2
2022-11-23T02:32:25.8114370Z dist init r=0, world=2
2022-11-23T02:32:25.8114591Z ok (3.311s)
2022-11-23T02:32:25.8115098Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8115778Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70313
2022-11-23T02:32:25.8116303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70314
2022-11-23T02:32:25.8116964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8117429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8118012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8118484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8119053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8119497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8120064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8120508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8120963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8121459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8122109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8122778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8123303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8123771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8124833Z dist init r=1, world=2
2022-11-23T02:32:25.8125073Z dist init r=0, world=2
2022-11-23T02:32:25.8125315Z ok (3.314s)
2022-11-23T02:32:25.8125827Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8126496Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70392
2022-11-23T02:32:25.8127028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70393
2022-11-23T02:32:25.8127715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8128170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8128724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8129191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8129769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8130199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8130770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8131236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8131963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8132450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8133116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8133807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8134332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8134866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8135231Z dist init r=0, world=2
2022-11-23T02:32:25.8135484Z dist init r=1, world=2
2022-11-23T02:32:25.8135710Z ok (3.411s)
2022-11-23T02:32:25.8136217Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8136907Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70471
2022-11-23T02:32:25.8137442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70472
2022-11-23T02:32:25.8138046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8138500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8139078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8139550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8140112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8140559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8141132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8141580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8142039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8142535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8143195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8143868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8144390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8144947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8145288Z dist init r=0, world=2
2022-11-23T02:32:25.8145544Z dist init r=1, world=2
2022-11-23T02:32:25.8145785Z ok (3.311s)
2022-11-23T02:32:25.8146294Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8146961Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70550
2022-11-23T02:32:25.8147490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70551
2022-11-23T02:32:25.8148106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8148557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8149119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8149590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8150168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8150596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8151167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8151687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8152154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8152636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8153296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8153994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8154516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8154971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8155322Z dist init r=0, world=2
2022-11-23T02:32:25.8155573Z dist init r=1, world=2
2022-11-23T02:32:25.8155803Z ok (3.411s)
2022-11-23T02:32:25.8156313Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8156993Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70629
2022-11-23T02:32:25.8157528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70630
2022-11-23T02:32:25.8158124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8158575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8159139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8159566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8160147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8160614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8161201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8161715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8162171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8162671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8163315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8164004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8164526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8164995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8165337Z dist init r=1, world=2
2022-11-23T02:32:25.8165595Z dist init r=0, world=2
2022-11-23T02:32:25.8165842Z ok (3.310s)
2022-11-23T02:32:25.8166335Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8167020Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70708
2022-11-23T02:32:25.8167550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70709
2022-11-23T02:32:25.8168211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8168654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8169237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8169709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8170293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8170721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8171291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8171948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8172390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8172892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8173556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8174246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8174754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8175232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8175584Z dist init r=1, world=2
2022-11-23T02:32:25.8175839Z dist init r=0, world=2
2022-11-23T02:32:25.8176063Z ok (3.310s)
2022-11-23T02:32:25.8176576Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8177263Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70787
2022-11-23T02:32:25.8177775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70788
2022-11-23T02:32:25.8178388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8178931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8179513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8180011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8180591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8181045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8181622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8182074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8182530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8183036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8183677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8184368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8184892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8185443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8185800Z dist init r=1, world=2
2022-11-23T02:32:25.8186054Z dist init r=0, world=2
2022-11-23T02:32:25.8186296Z ok (3.311s)
2022-11-23T02:32:25.8186783Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8187469Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70866
2022-11-23T02:32:25.8188000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70867
2022-11-23T02:32:25.8188615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8189053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8189631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8190106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8190689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8191120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8191696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8192160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8192602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8193100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8193764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8194453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8194960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8195430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8195856Z dist init r=0, world=2
2022-11-23T02:32:25.8196093Z dist init r=1, world=2
2022-11-23T02:32:25.8196329Z ok (3.411s)
2022-11-23T02:32:25.8196839Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8197519Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70945
2022-11-23T02:32:25.8198036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70946
2022-11-23T02:32:25.8198650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8199104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8199678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8200137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8200715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8201162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8201715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8202237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8202703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8203203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8203845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8204539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8205060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8205530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8205871Z dist init r=1, world=2
2022-11-23T02:32:25.8206126Z dist init r=0, world=2
2022-11-23T02:32:25.8206369Z ok (3.310s)
2022-11-23T02:32:25.8206862Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8207537Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71024
2022-11-23T02:32:25.8208072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71025
2022-11-23T02:32:25.8208685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8209119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8209695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8210166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8210735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8211186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8212017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8212489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8213020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8213519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8214182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8214871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8215382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8215857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8216214Z dist init r=0, world=2
2022-11-23T02:32:25.8216450Z dist init r=1, world=2
2022-11-23T02:32:25.8216693Z ok (3.311s)
2022-11-23T02:32:25.8217204Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8217883Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71103
2022-11-23T02:32:25.8218398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71104
2022-11-23T02:32:25.8219092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8219562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8220120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8220596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8221178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8221623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8222176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8222644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8223101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8223602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8224250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8224941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8225476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8225929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8226287Z dist init r=0, world=2
2022-11-23T02:32:25.8226541Z dist init r=1, world=2
2022-11-23T02:32:25.8226784Z ok (3.411s)
2022-11-23T02:32:25.8227274Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8227957Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71182
2022-11-23T02:32:25.8228487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71183
2022-11-23T02:32:25.8229082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8229599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8230177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8230652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8231217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8231664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8232238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8232707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8233145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8233649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8234309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8234980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8235506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8236032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8236401Z dist init r=1, world=2
2022-11-23T02:32:25.8236637Z dist init r=0, world=2
2022-11-23T02:32:25.8236878Z ok (3.311s)
2022-11-23T02:32:25.8237352Z   test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71261
2022-11-23T02:32:25.8237906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71262
2022-11-23T02:32:25.8238518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8238968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8239543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8240000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8240582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8241024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8241578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8242048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8242510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8243009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8243653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8244342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8244869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8245346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8246597Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8247515Z   warnings.warn(
2022-11-23T02:32:25.8248679Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8249455Z   warnings.warn(
2022-11-23T02:32:25.8249709Z dist init r=0, world=2
2022-11-23T02:32:25.8249944Z dist init r=1, world=2
2022-11-23T02:32:25.8250184Z ok (3.209s)
2022-11-23T02:32:25.8250664Z   test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71340
2022-11-23T02:32:25.8251237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71341
2022-11-23T02:32:25.8252027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8252485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8253143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8253630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8254194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8254642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8255223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8255678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8256137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8256636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8257301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8257976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8258500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8258974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8260243Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8261005Z   warnings.warn(
2022-11-23T02:32:25.8262162Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8284006Z   warnings.warn(
2022-11-23T02:32:25.8284266Z dist init r=0, world=2
2022-11-23T02:32:25.8284528Z dist init r=1, world=2
2022-11-23T02:32:25.8284769Z ok (3.410s)
2022-11-23T02:32:25.8285235Z   test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71419
2022-11-23T02:32:25.8285803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71420
2022-11-23T02:32:25.8286460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8286925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8287491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8287969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8288558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8289004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8289554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8290017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8290475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8291065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8291980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8292387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8292627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8292859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8293884Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8294001Z   warnings.warn(
2022-11-23T02:32:25.8295001Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8295120Z   warnings.warn(
2022-11-23T02:32:25.8295233Z dist init r=0, world=2
2022-11-23T02:32:25.8295326Z dist init r=1, world=2
2022-11-23T02:32:25.8295428Z ok (3.310s)
2022-11-23T02:32:25.8295777Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8296225Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71498
2022-11-23T02:32:25.8296449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71499
2022-11-23T02:32:25.8296821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8297115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8297503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8297678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8298053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8298231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8298604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8298795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8299048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8299298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8299698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8300092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8300305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8300604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8301244Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8301361Z   warnings.warn(
2022-11-23T02:32:25.8301988Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8302102Z   warnings.warn(
2022-11-23T02:32:25.8302218Z dist init r=1, world=2
2022-11-23T02:32:25.8302327Z dist init r=0, world=2
2022-11-23T02:32:25.8302410Z ok (3.310s)
2022-11-23T02:32:25.8302757Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8303207Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71577
2022-11-23T02:32:25.8303430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71578
2022-11-23T02:32:25.8303801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8303984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8304363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8304555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8304922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8305082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8305466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8305661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8305908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8306217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8306623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8307021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8307254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8307490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8308095Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8308214Z   warnings.warn(
2022-11-23T02:32:25.8308835Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8308947Z   warnings.warn(
2022-11-23T02:32:25.8309062Z dist init r=1, world=2
2022-11-23T02:32:25.8309178Z dist init r=0, world=2
2022-11-23T02:32:25.8309280Z ok (3.310s)
2022-11-23T02:32:25.8309611Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8310107Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71656
2022-11-23T02:32:25.8310318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71657
2022-11-23T02:32:25.8310695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8310878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8311260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8311452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8311819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8311996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8312377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8312552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8312801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8313051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8313454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8313852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8314083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8314315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8314934Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8315049Z   warnings.warn(
2022-11-23T02:32:25.8315670Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8315820Z   warnings.warn(
2022-11-23T02:32:25.8315935Z dist init r=0, world=2
2022-11-23T02:32:25.8316046Z dist init r=1, world=2
2022-11-23T02:32:25.8316150Z ok (3.310s)
2022-11-23T02:32:25.8316478Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8316933Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71735
2022-11-23T02:32:25.8317157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71736
2022-11-23T02:32:25.8317529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8317693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8318076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8318274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8318637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8318813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8319254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8319455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8319708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8319955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8320346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8320741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8320973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8321204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8321828Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8321943Z   warnings.warn(
2022-11-23T02:32:25.8322556Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8322670Z   warnings.warn(
2022-11-23T02:32:25.8322782Z dist init r=0, world=2
2022-11-23T02:32:25.8322875Z dist init r=1, world=2
2022-11-23T02:32:25.8322975Z ok (3.310s)
2022-11-23T02:32:25.8323306Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8323756Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71814
2022-11-23T02:32:25.8323977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71815
2022-11-23T02:32:25.8324347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8324617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8324998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8325172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8325542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8325720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8326094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8326284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8326530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8326776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8327175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8327572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8327786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8328011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8328681Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8328804Z   warnings.warn(
2022-11-23T02:32:25.8329426Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8329545Z   warnings.warn(
2022-11-23T02:32:25.8329658Z dist init r=0, world=2
2022-11-23T02:32:25.8329770Z dist init r=1, world=2
2022-11-23T02:32:25.8329852Z ok (3.310s)
2022-11-23T02:32:25.8330187Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8330640Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71893
2022-11-23T02:32:25.8330861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71894
2022-11-23T02:32:25.8331232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8331408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8332048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8332246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8332618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8332775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8333154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8333347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8333594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8333839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8334342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8334743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8334975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8335207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8335808Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8335926Z   warnings.warn(
2022-11-23T02:32:25.8336544Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8336657Z   warnings.warn(
2022-11-23T02:32:25.8336769Z dist init r=0, world=2
2022-11-23T02:32:25.8336880Z dist init r=1, world=2
2022-11-23T02:32:25.8336980Z ok (3.310s)
2022-11-23T02:32:25.8337333Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8337855Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71972
2022-11-23T02:32:25.8338069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71973
2022-11-23T02:32:25.8338443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8338622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8339007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8339200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8339568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8339742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8340117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8340308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8340538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8340784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8341187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8341585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8341818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8342039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8342658Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8342773Z   warnings.warn(
2022-11-23T02:32:25.8343388Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8343537Z   warnings.warn(
2022-11-23T02:32:25.8343652Z dist init r=0, world=2
2022-11-23T02:32:25.8343763Z dist init r=1, world=2
2022-11-23T02:32:25.8343924Z ok (3.410s)
2022-11-23T02:32:25.8344280Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8344740Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72051
2022-11-23T02:32:25.8344965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72052
2022-11-23T02:32:25.8345335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8345495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8345879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8346070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8346438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8346613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8347028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8347225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8347471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8347714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8348104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8348502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8348733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8348964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8349585Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8349701Z   warnings.warn(
2022-11-23T02:32:25.8350321Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8350436Z   warnings.warn(
2022-11-23T02:32:25.8350547Z dist init r=1, world=2
2022-11-23T02:32:25.8350639Z dist init r=0, world=2
2022-11-23T02:32:25.8350739Z ok (3.310s)
2022-11-23T02:32:25.8351073Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8351525Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72130
2022-11-23T02:32:25.8351747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72131
2022-11-23T02:32:25.8352117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8352293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8352733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8352909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8353277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8353451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8353823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8354021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8354269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8354513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8354916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8355318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8355531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8355760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8356427Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8356551Z   warnings.warn(
2022-11-23T02:32:25.8357171Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8357286Z   warnings.warn(
2022-11-23T02:32:25.8357398Z dist init r=0, world=2
2022-11-23T02:32:25.8357508Z dist init r=1, world=2
2022-11-23T02:32:25.8357610Z ok (3.410s)
2022-11-23T02:32:25.8357925Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8358373Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72209
2022-11-23T02:32:25.8358595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72210
2022-11-23T02:32:25.8358965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8359139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8359518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8359713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8360077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8360231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8360606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8360797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8361043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8361285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8361687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8362146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8362380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8362611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8363232Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8363329Z   warnings.warn(
2022-11-23T02:32:25.8363944Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8364055Z   warnings.warn(
2022-11-23T02:32:25.8364169Z dist init r=0, world=2
2022-11-23T02:32:25.8364277Z dist init r=1, world=2
2022-11-23T02:32:25.8364376Z ok (3.310s)
2022-11-23T02:32:25.8364711Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8365158Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72288
2022-11-23T02:32:25.8365408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72289
2022-11-23T02:32:25.8365787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8365965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8366345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8366542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8366910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8367084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8367457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8367649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8367880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8368127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8368526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8368924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8369144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8369372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8369484Z dist init r=1, world=2
2022-11-23T02:32:25.8369592Z dist init r=0, world=2
2022-11-23T02:32:25.8369673Z ok (3.310s)
2022-11-23T02:32:25.8370010Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8370461Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72367
2022-11-23T02:32:25.8370680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72368
2022-11-23T02:32:25.8371111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8371291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8371850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8372031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8372418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8372592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8372966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8373157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8373404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8373647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8374046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8374439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8374739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8374980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8375076Z dist init r=0, world=2
2022-11-23T02:32:25.8375189Z dist init r=1, world=2
2022-11-23T02:32:25.8375289Z ok (3.310s)
2022-11-23T02:32:25.8375615Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8376067Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72446
2022-11-23T02:32:25.8376288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72447
2022-11-23T02:32:25.8376659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8376836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8377203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8377397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8377762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8377940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8378313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8378502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8378748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8378991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8379378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8379775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8380050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8380351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8380465Z dist init r=1, world=2
2022-11-23T02:32:25.8380577Z dist init r=0, world=2
2022-11-23T02:32:25.8380680Z ok (3.310s)
2022-11-23T02:32:25.8381002Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8381458Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72525
2022-11-23T02:32:25.8381661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72526
2022-11-23T02:32:25.8382032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8382207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8382589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8382780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8383145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8383320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8383691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8383926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8384182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8384428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8384834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8385236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8385467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8385697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8385810Z dist init r=1, world=2
2022-11-23T02:32:25.8385919Z dist init r=0, world=2
2022-11-23T02:32:25.8386003Z ok (3.310s)
2022-11-23T02:32:25.8386331Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8386779Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72604
2022-11-23T02:32:25.8387002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72605
2022-11-23T02:32:25.8387371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8387547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8387925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8388118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8388474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8388650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8389023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8389211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8389515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8389761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8390165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8390559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8390794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8391004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8391119Z dist init r=1, world=2
2022-11-23T02:32:25.8391230Z dist init r=0, world=2
2022-11-23T02:32:25.8391330Z ok (3.310s)
2022-11-23T02:32:25.8391653Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8392103Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72683
2022-11-23T02:32:25.8392322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72684
2022-11-23T02:32:25.8392690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8392893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8393285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8393478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8393845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8394024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8394396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8394587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8394832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8395079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8395461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8395852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8396080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8396308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8396421Z dist init r=1, world=2
2022-11-23T02:32:25.8396530Z dist init r=0, world=2
2022-11-23T02:32:25.8396630Z ok (3.310s)
2022-11-23T02:32:25.8396968Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8397401Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72762
2022-11-23T02:32:25.8397621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72763
2022-11-23T02:32:25.8397990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8398166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8398602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8398796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8399162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8399336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8399709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8399879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8400126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8400372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8400775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8401168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8401396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8401624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8401737Z dist init r=1, world=2
2022-11-23T02:32:25.8401871Z dist init r=0, world=2
2022-11-23T02:32:25.8401979Z ok (3.310s)
2022-11-23T02:32:25.8402318Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8402768Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72841
2022-11-23T02:32:25.8402993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72842
2022-11-23T02:32:25.8403363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8403542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8403923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8404117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8404468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8404645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8405019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8405211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8405456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8405700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8406101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8406499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8406728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8406939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8407052Z dist init r=0, world=2
2022-11-23T02:32:25.8407160Z dist init r=1, world=2
2022-11-23T02:32:25.8407313Z ok (3.310s)
2022-11-23T02:32:25.8407640Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8408091Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72920
2022-11-23T02:32:25.8408314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72921
2022-11-23T02:32:25.8408686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8408846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8409226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8409417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8409788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8409962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8410334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8410524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8410768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8411041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8411758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8412177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8412417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8412642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8412757Z dist init r=0, world=2
2022-11-23T02:32:25.8412867Z dist init r=1, world=2
2022-11-23T02:32:25.8412966Z ok (3.310s)
2022-11-23T02:32:25.8413270Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8413723Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72999
2022-11-23T02:32:25.8413945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73000
2022-11-23T02:32:25.8414317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8414495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8414874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8415066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8415434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8415606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8415960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8416150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8416399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8416643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8417144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8417543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8417773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8418003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8418121Z dist init r=0, world=2
2022-11-23T02:32:25.8418215Z dist init r=1, world=2
2022-11-23T02:32:25.8418316Z ok (3.310s)
2022-11-23T02:32:25.8418644Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8418961Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73078
2022-11-23T02:32:25.8419183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73079
2022-11-23T02:32:25.8419559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8419734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8420113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8420350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8420732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8420908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8421279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8421471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8421715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8421965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8422361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8422760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8422971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8423200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8423312Z dist init r=1, world=2
2022-11-23T02:32:25.8423423Z dist init r=0, world=2
2022-11-23T02:32:25.8423525Z ok (3.812s)
2022-11-23T02:32:25.8423851Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8424171Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73161
2022-11-23T02:32:25.8424390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73162
2022-11-23T02:32:25.8424747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8424923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8425305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8425495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8425921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8426098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8426472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8426662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8426912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8427139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8427538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8427931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8428165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8428390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8428503Z dist init r=0, world=2
2022-11-23T02:32:25.8428611Z dist init r=1, world=2
2022-11-23T02:32:25.8428711Z ok (3.311s)
2022-11-23T02:32:25.8429070Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8429400Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73240
2022-11-23T02:32:25.8429619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73241
2022-11-23T02:32:25.8429995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8430175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8430556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8430748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8431115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8431293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8431649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8431838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8432082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8432328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8432725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8433119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8433347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8433577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8433690Z dist init r=1, world=2
2022-11-23T02:32:25.8433784Z dist init r=0, world=2
2022-11-23T02:32:25.8433883Z ok (3.812s)
2022-11-23T02:32:25.8434205Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8434581Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73323
2022-11-23T02:32:25.8434802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73324
2022-11-23T02:32:25.8435175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8435351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8435731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8435905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8436273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8436446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8436828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8437017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8437263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8437500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8437934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8438327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8438540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8438755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8438865Z dist init r=1, world=2
2022-11-23T02:32:25.8438978Z dist init r=0, world=2
2022-11-23T02:32:25.8439078Z ok (3.311s)
2022-11-23T02:32:25.8439405Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8439715Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73402
2022-11-23T02:32:25.8439935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73403
2022-11-23T02:32:25.8440286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8440458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8440829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8441016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8441380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8441555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8441928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8442117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8442368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8442594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8442993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8443452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8443684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8443914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8444030Z dist init r=0, world=2
2022-11-23T02:32:25.8444140Z dist init r=1, world=2
2022-11-23T02:32:25.8444241Z ok (3.812s)
2022-11-23T02:32:25.8444557Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8444882Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73485
2022-11-23T02:32:25.8445102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73486
2022-11-23T02:32:25.8445479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8445656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8446039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8446232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8446598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8446836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8447205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8447402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8447648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8447896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8448296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8448691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8448920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8449150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8449246Z dist init r=0, world=2
2022-11-23T02:32:25.8449356Z dist init r=1, world=2
2022-11-23T02:32:25.8449456Z ok (3.311s)
2022-11-23T02:32:25.8449787Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8450108Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73564
2022-11-23T02:32:25.8450327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73565
2022-11-23T02:32:25.8450700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8450876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8451260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8451663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8452056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8452321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8452700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8452891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8453139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8453385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8453790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8454167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8454397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8454630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8454744Z dist init r=1, world=2
2022-11-23T02:32:25.8454855Z dist init r=0, world=2
2022-11-23T02:32:25.8454956Z ok (3.812s)
2022-11-23T02:32:25.8455284Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8455600Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73647
2022-11-23T02:32:25.8455880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73648
2022-11-23T02:32:25.8456251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8456432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8456814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8457013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8457380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8457555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8457927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8458120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8458349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8458596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8458998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8459396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8459625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8459850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8459963Z dist init r=0, world=2
2022-11-23T02:32:25.8460072Z dist init r=1, world=2
2022-11-23T02:32:25.8460154Z ok (3.210s)
2022-11-23T02:32:25.8460479Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8460800Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73726
2022-11-23T02:32:25.8461016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73727
2022-11-23T02:32:25.8461451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8461631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8462012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8462207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8462576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8462733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8463109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8463299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8463547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8463790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8464190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8464584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8464896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8465135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8465230Z dist init r=0, world=2
2022-11-23T02:32:25.8465342Z dist init r=1, world=2
2022-11-23T02:32:25.8465444Z ok (3.812s)
2022-11-23T02:32:25.8465766Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8466090Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73809
2022-11-23T02:32:25.8466311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73810
2022-11-23T02:32:25.8466683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8466862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8467223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8467418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8467790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8467966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8468339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8468528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8468775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8469020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8469418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8469798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8470031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8470316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8470431Z dist init r=0, world=2
2022-11-23T02:32:25.8470544Z dist init r=1, world=2
2022-11-23T02:32:25.8470647Z ok (3.812s)
2022-11-23T02:32:25.8470970Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8471290Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73892
2022-11-23T02:32:25.8471492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73893
2022-11-23T02:32:25.8471866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8472044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8472424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8472618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8472988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8473164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8473581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8473780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8474007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8474254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8474658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8475055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8475285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8475514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8475630Z dist init r=1, world=2
2022-11-23T02:32:25.8475739Z dist init r=0, world=2
2022-11-23T02:32:25.8475825Z ok (3.811s)
2022-11-23T02:32:25.8476145Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8476461Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73975
2022-11-23T02:32:25.8476682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73976
2022-11-23T02:32:25.8477056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8477233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8477614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8477806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8478175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8478331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8478703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8478944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8479191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8479435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8479835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8480286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8480519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8480749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8480845Z dist init r=1, world=2
2022-11-23T02:32:25.8480955Z dist init r=0, world=2
2022-11-23T02:32:25.8481054Z ok (3.812s)
2022-11-23T02:32:25.8481375Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8481804Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74058
2022-11-23T02:32:25.8482026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74059
2022-11-23T02:32:25.8482397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8482622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8482994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8483191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8483557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8483737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8484116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8484309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8484554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8484801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8485183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8485581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8485813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8486041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8486657Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8486773Z   warnings.warn(
2022-11-23T02:32:25.8487393Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8487504Z   warnings.warn(
2022-11-23T02:32:25.8487614Z dist init r=1, world=2
2022-11-23T02:32:25.8487706Z dist init r=0, world=2
2022-11-23T02:32:25.8487807Z ok (3.812s)
2022-11-23T02:32:25.8488119Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8488607Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74141
2022-11-23T02:32:25.8488830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74142
2022-11-23T02:32:25.8489199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8489379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8489758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8489953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8490302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8490480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8490857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8491048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8491292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8491771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8492263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8492676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8492910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8493127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8494148Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8494268Z   warnings.warn(
2022-11-23T02:32:25.8495264Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8495377Z   warnings.warn(
2022-11-23T02:32:25.8496193Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8496307Z   warnings.warn(
2022-11-23T02:32:25.8496928Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8497043Z   warnings.warn(
2022-11-23T02:32:25.8497159Z dist init r=1, world=2
2022-11-23T02:32:25.8497273Z dist init r=0, world=2
2022-11-23T02:32:25.8497355Z ok (3.811s)
2022-11-23T02:32:25.8497671Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8498181Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74224
2022-11-23T02:32:25.8498404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74225
2022-11-23T02:32:25.8498776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8498959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8499339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8499535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8499883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8500064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8500439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8500628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8500874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8501119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8501566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8501972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8502206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8502418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8502534Z dist init r=1, world=2
2022-11-23T02:32:25.8502644Z dist init r=0, world=2
2022-11-23T02:32:25.8502743Z ok (3.210s)
2022-11-23T02:32:25.8503059Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8503494Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74303
2022-11-23T02:32:25.8503714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74304
2022-11-23T02:32:25.8504085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8504244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8504630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8504821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8505185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8505358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8505729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8505921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8506166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8506410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8506791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8507246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8507478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8507710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8507826Z dist init r=1, world=2
2022-11-23T02:32:25.8507938Z dist init r=0, world=2
2022-11-23T02:32:25.8508042Z ok (3.210s)
2022-11-23T02:32:25.8508353Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8508763Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74382
2022-11-23T02:32:25.8508990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74383
2022-11-23T02:32:25.8509357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8509533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8509912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8510105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8510530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8510714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8511092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8511267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8511518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8511762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8512162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8512558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8512789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8513017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8513133Z dist init r=1, world=2
2022-11-23T02:32:25.8513227Z dist init r=0, world=2
2022-11-23T02:32:25.8513329Z ok (3.812s)
2022-11-23T02:32:25.8513637Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8514067Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74465
2022-11-23T02:32:25.8514286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74466
2022-11-23T02:32:25.8514655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8514835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8515209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8515402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8515748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8515978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8516349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8516540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8516790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8517038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8517438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8517837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8518047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8518283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8519341Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8519463Z   warnings.warn(
2022-11-23T02:32:25.8520462Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8520582Z   warnings.warn(
2022-11-23T02:32:25.8520696Z dist init r=1, world=2
2022-11-23T02:32:25.8520809Z dist init r=0, world=2
2022-11-23T02:32:25.8520912Z ok (3.812s)
2022-11-23T02:32:25.8521214Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8521644Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74548
2022-11-23T02:32:25.8521848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74549
2022-11-23T02:32:25.8522217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8522394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8522776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8522970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8523339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8523519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8523896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8524087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8524316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8524561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8525021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8525419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8525650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8525884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8525999Z dist init r=0, world=2
2022-11-23T02:32:25.8526112Z dist init r=1, world=2
2022-11-23T02:32:25.8526195Z ok (3.912s)
2022-11-23T02:32:25.8526497Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8526924Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74631
2022-11-23T02:32:25.8527147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74632
2022-11-23T02:32:25.8527519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8527696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8528075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8528268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8528663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8528849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8529228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8529426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8529674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8529923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8530321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8530718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8530948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8531156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8532409Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8532534Z   warnings.warn(
2022-11-23T02:32:25.8533540Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8533654Z   warnings.warn(
2022-11-23T02:32:25.8533772Z dist init r=1, world=2
2022-11-23T02:32:25.8533970Z dist init r=0, world=2
2022-11-23T02:32:25.8534070Z ok (3.712s)
2022-11-23T02:32:25.8534307Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8534618Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74714
2022-11-23T02:32:25.8534838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74715
2022-11-23T02:32:25.8535194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8535376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8535756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8535950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8536320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8536500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8536876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8537063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8537292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8537600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8538018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8538416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8538651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8538883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8538998Z dist init r=1, world=2
2022-11-23T02:32:25.8539109Z dist init r=0, world=2
2022-11-23T02:32:25.8539190Z ok (3.511s)
2022-11-23T02:32:25.8539431Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8539747Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74793
2022-11-23T02:32:25.8539968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74794
2022-11-23T02:32:25.8540340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8540517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8540901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8541091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8541453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8541611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8541982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8542171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8542418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8542662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8543063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8543519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8543752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8543982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8544077Z dist init r=1, world=2
2022-11-23T02:32:25.8544187Z dist init r=0, world=2
2022-11-23T02:32:25.8544290Z ok (3.511s)
2022-11-23T02:32:25.8544644Z   test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74872
2022-11-23T02:32:25.8544864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74873
2022-11-23T02:32:25.8545236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8545417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8545796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8545971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8546337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8546558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8546942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8547133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8547384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8547634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8548032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8548427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8548639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8548869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8549888Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8550008Z   warnings.warn(
2022-11-23T02:32:25.8551006Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8551111Z   warnings.warn(
2022-11-23T02:32:25.8551349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8551587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8551824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8552112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8552208Z dist init r=1, world=2
2022-11-23T02:32:25.8552323Z dist init r=0, world=2
2022-11-23T02:32:25.8552427Z ok (3.910s)
2022-11-23T02:32:25.8552780Z   test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74955
2022-11-23T02:32:25.8553005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74956
2022-11-23T02:32:25.8553386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8553566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8553949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8554150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8554498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8554675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8555055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8555245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8555537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8555791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8556195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8556599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8556811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8557041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8558060Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8558175Z   warnings.warn(
2022-11-23T02:32:25.8559171Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8559288Z   warnings.warn(
2022-11-23T02:32:25.8559528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8559758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8559992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8560221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8560336Z dist init r=1, world=2
2022-11-23T02:32:25.8560431Z dist init r=0, world=2
2022-11-23T02:32:25.8560590Z ok (3.811s)
2022-11-23T02:32:25.8560934Z   test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75038
2022-11-23T02:32:25.8561157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75039
2022-11-23T02:32:25.8561536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8561714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8562098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8562292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8562637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8562811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8563193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8563384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8563631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8563877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8564322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8564731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8564963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8565175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8566196Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8566315Z   warnings.warn(
2022-11-23T02:32:25.8567314Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8567428Z   warnings.warn(
2022-11-23T02:32:25.8567664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8567899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8568132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8568367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:32:25.8568485Z dist init r=0, world=2
2022-11-23T02:32:25.8568597Z dist init r=1, world=2
2022-11-23T02:32:25.8568682Z ok (3.810s)
2022-11-23T02:32:25.8569051Z   test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75121
2022-11-23T02:32:25.8569271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75122
2022-11-23T02:32:25.8569706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8569887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8570267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8570465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8570834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8570988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8571368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8571784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8572045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8572296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8572704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8573101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8573417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8573661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8573756Z dist init r=1, world=2
2022-11-23T02:32:25.8573871Z dist init r=0, world=2
2022-11-23T02:32:25.8573974Z ok (3.813s)
2022-11-23T02:32:25.8574350Z   test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75204
2022-11-23T02:32:25.8574573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75205
2022-11-23T02:32:25.8574948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8575128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8575512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8575684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8576055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8576228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8576601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8576794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8577039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8577282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8577679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8578079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8578292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8578520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8579226Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8579342Z   warnings.warn(
2022-11-23T02:32:25.8579996Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8580107Z   warnings.warn(
2022-11-23T02:32:25.8580215Z dist init r=0, world=2
2022-11-23T02:32:25.8580323Z dist init r=1, world=2
2022-11-23T02:32:25.8580405Z ok (3.812s)
2022-11-23T02:32:25.8580767Z   test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75287
2022-11-23T02:32:25.8580990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75288
2022-11-23T02:32:25.8581366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8581540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8581919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8582115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8582533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8582719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8583076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8583261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8583511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8583753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8584154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8584550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8584783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8585011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8585121Z dist init r=1, world=2
2022-11-23T02:32:25.8585214Z dist init r=0, world=2
2022-11-23T02:32:25.8585316Z ok (3.812s)
2022-11-23T02:32:25.8585610Z   test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75370
2022-11-23T02:32:25.8585831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75371
2022-11-23T02:32:25.8586199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8586373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8586738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8586915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8587281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8587473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8587847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8588102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8588349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8588594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8588993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8589393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8589604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8589827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8589940Z dist init r=0, world=2
2022-11-23T02:32:25.8590055Z dist init r=1, world=2
2022-11-23T02:32:25.8590152Z ok (3.310s)
2022-11-23T02:32:25.8590554Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75449
2022-11-23T02:32:25.8590771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75450
2022-11-23T02:32:25.8591143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8591361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8591731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8591924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8592292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8592466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8592832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8593014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8593255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8593503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8593884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8594280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8594506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8594736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8595754Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8595866Z   warnings.warn(
2022-11-23T02:32:25.8596874Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8597037Z   warnings.warn(
2022-11-23T02:32:25.8597650Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8597752Z   warnings.warn(
2022-11-23T02:32:25.8598369Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8598480Z   warnings.warn(
2022-11-23T02:32:25.8598574Z dist init r=1, world=2
2022-11-23T02:32:25.8598681Z dist init r=0, world=2
2022-11-23T02:32:25.8598781Z ok (3.311s)
2022-11-23T02:32:25.8599178Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75528
2022-11-23T02:32:25.8599390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75529
2022-11-23T02:32:25.8599754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8599924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8600349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8600532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8600893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8601060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8601430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8601614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8601848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8602085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8602483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8602861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8603074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8603295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8604007Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:32:25.8604114Z   warnings.warn(
2022-11-23T02:32:25.8604822Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:32:25.8604924Z   warnings.warn(
2022-11-23T02:32:25.8605030Z dist init r=1, world=2
2022-11-23T02:32:25.8605132Z dist init r=0, world=2
2022-11-23T02:32:25.8605225Z ok (3.311s)
2022-11-23T02:32:25.8605606Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75607
2022-11-23T02:32:25.8605909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75608
2022-11-23T02:32:25.8606278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8606442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8606815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8607003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8607361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8607534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8607906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8608075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8608314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8608556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8608994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8609393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8609619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8609841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8610860Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8610968Z   warnings.warn(
2022-11-23T02:32:25.8612208Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8612325Z   warnings.warn(
2022-11-23T02:32:25.8612932Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8613040Z   warnings.warn(
2022-11-23T02:32:25.8613652Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:32:25.8613758Z   warnings.warn(
2022-11-23T02:32:25.8613863Z dist init r=1, world=2
2022-11-23T02:32:25.8613970Z dist init r=0, world=2
2022-11-23T02:32:25.8614065Z ok (3.311s)
2022-11-23T02:32:25.8614460Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75686
2022-11-23T02:32:25.8614765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75687
2022-11-23T02:32:25.8615125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8615296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8615674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8615865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8616222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8616395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8616761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8616944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8617175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8617415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8617811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8618262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8618503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8618727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8619435Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:32:25.8619555Z   warnings.warn(
2022-11-23T02:32:25.8620261Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:32:25.8620369Z   warnings.warn(
2022-11-23T02:32:25.8620464Z dist init r=0, world=2
2022-11-23T02:32:25.8620575Z dist init r=1, world=2
2022-11-23T02:32:25.8620668Z ok (3.312s)
2022-11-23T02:32:25.8621048Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75765
2022-11-23T02:32:25.8621269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75766
2022-11-23T02:32:25.8621641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8621811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8622182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8622356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8622717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8622884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8623253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8623492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8623734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8623982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8624376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8624766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8624978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8625202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8626209Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8626325Z   warnings.warn(
2022-11-23T02:32:25.8627358Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8627472Z   warnings.warn(
2022-11-23T02:32:25.8627578Z dist init r=0, world=2
2022-11-23T02:32:25.8627685Z dist init r=1, world=2
2022-11-23T02:32:25.8627784Z ok (3.312s)
2022-11-23T02:32:25.8628173Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75844
2022-11-23T02:32:25.8628391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75845
2022-11-23T02:32:25.8628750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8628923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8629295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8629478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8629831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8630005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8630370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8630561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8630789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8631029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8631427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8631823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8632048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8632316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8633026Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:32:25.8633135Z   warnings.warn(
2022-11-23T02:32:25.8633845Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:32:25.8633951Z   warnings.warn(
2022-11-23T02:32:25.8634045Z dist init r=1, world=2
2022-11-23T02:32:25.8634148Z dist init r=0, world=2
2022-11-23T02:32:25.8634241Z ok (3.211s)
2022-11-23T02:32:25.8634627Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75923
2022-11-23T02:32:25.8634840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75924
2022-11-23T02:32:25.8635204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8635381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8635815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8635995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8636354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8636520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8636886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8637067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8637309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8637552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8637951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8638342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8638554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8638781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8639793Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8639902Z   warnings.warn(
2022-11-23T02:32:25.8640903Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8641066Z   warnings.warn(
2022-11-23T02:32:25.8641175Z dist init r=1, world=2
2022-11-23T02:32:25.8641281Z dist init r=0, world=2
2022-11-23T02:32:25.8641383Z ok (3.311s)
2022-11-23T02:32:25.8641766Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76002
2022-11-23T02:32:25.8641980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76003
2022-11-23T02:32:25.8642342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8642515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8642885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8643075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8643438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8643612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8644032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8644215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8644496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8644743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8645140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8645536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8645764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8645983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8646692Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:32:25.8646801Z   warnings.warn(
2022-11-23T02:32:25.8647508Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2022-11-23T02:32:25.8647616Z   warnings.warn(
2022-11-23T02:32:25.8647711Z dist init r=1, world=2
2022-11-23T02:32:25.8647814Z dist init r=0, world=2
2022-11-23T02:32:25.8647915Z ok (3.211s)
2022-11-23T02:32:25.8648195Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8648508Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76081
2022-11-23T02:32:25.8648728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76082
2022-11-23T02:32:25.8649092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8649270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8649633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8649872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8650233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8650407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8650771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8650964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8651209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8651678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8652094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8652478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8652702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8652927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8653036Z dist init r=1, world=2
2022-11-23T02:32:25.8653139Z dist init r=0, world=2
2022-11-23T02:32:25.8653236Z ok (3.511s)
2022-11-23T02:32:25.8653592Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8653919Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76160
2022-11-23T02:32:25.8654122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76161
2022-11-23T02:32:25.8654500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8654675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8655045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8655236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8655595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8655767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8656130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8656318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8656544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8656782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8657173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8657567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8657793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8658018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8658131Z dist init r=0, world=2
2022-11-23T02:32:25.8658234Z dist init r=1, world=2
2022-11-23T02:32:25.8658316Z ok (3.210s)
2022-11-23T02:32:25.8658589Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8658988Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76239
2022-11-23T02:32:25.8659204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76240
2022-11-23T02:32:25.8659575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8659748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8660123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8660310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8660664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8660819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8661190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8661378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8661617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8661854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8662290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8662687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8662911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8663115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8663226Z dist init r=0, world=2
2022-11-23T02:32:25.8663334Z dist init r=1, world=2
2022-11-23T02:32:25.8663427Z ok (3.411s)
2022-11-23T02:32:25.8663697Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T02:32:25.8664007Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76318
2022-11-23T02:32:25.8664221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76319
2022-11-23T02:32:25.8664586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8664744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8665115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8665307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8665663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8665829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8666195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8666374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8666621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8666857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8667239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8667688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8667916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8668145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8668257Z dist init r=0, world=2
2022-11-23T02:32:25.8668366Z dist init r=1, world=2
2022-11-23T02:32:25.8668464Z ok (3.411s)
2022-11-23T02:32:25.8668828Z   test_state_dict_with_shared_parameters_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76397
2022-11-23T02:32:25.8669028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76398
2022-11-23T02:32:25.8669401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8669577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8669959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8670147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8670513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8670686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8671100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8671299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8671527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8671772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8672176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8672570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8672797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8673024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8673141Z dist init r=0, world=2
2022-11-23T02:32:25.8673254Z dist init r=1, world=2
2022-11-23T02:32:25.8673337Z ok (3.309s)
2022-11-23T02:32:25.8673702Z   test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76476
2022-11-23T02:32:25.8673921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76477
2022-11-23T02:32:25.8674297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8674471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8674851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8675040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8675404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8675578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8675933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8676123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8676426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8676671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8677064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8677455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8677685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8677909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8678004Z dist init r=0, world=2
2022-11-23T02:32:25.8678113Z dist init r=1, world=2
2022-11-23T02:32:25.8678210Z ok (3.409s)
2022-11-23T02:32:25.8678563Z   test_state_dict_with_shared_parameters_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76555
2022-11-23T02:32:25.8678785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76556
2022-11-23T02:32:25.8679158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8679331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8679710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8679995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8680357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8680533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8680907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8681102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8681348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8681586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8681982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8682381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8682592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8682819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8682929Z dist init r=0, world=2
2022-11-23T02:32:25.8683041Z dist init r=1, world=2
2022-11-23T02:32:25.8683140Z ok (3.410s)
2022-11-23T02:32:25.8683445Z   test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76634
2022-11-23T02:32:25.8683664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76635
2022-11-23T02:32:25.8684033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8684190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8684575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8684767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8685134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:25.8685367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:25.8685746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:25.8685939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:25.8686189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:25.8686434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:25.8686816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8687213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:25.8687441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:25.8687673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:25.8688685Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8688850Z   warnings.warn(
2022-11-23T02:32:25.8689857Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:32:25.8689973Z   warnings.warn(
2022-11-23T02:32:25.8690087Z dist init r=0, world=2
2022-11-23T02:32:25.8690195Z dist init r=1, world=2
2022-11-23T02:32:25.8690277Z ok (3.309s)
2022-11-23T02:32:25.8690318Z 
2022-11-23T02:32:25.8690570Z ----------------------------------------------------------------------
2022-11-23T02:32:25.8690690Z Ran 116 tests in 396.406s
2022-11-23T02:32:25.8690710Z 
2022-11-23T02:32:25.8690805Z OK
2022-11-23T02:32:25.8690824Z 
2022-11-23T02:32:25.8690952Z Generating XML reports...
2022-11-23T02:32:25.8691398Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123022548.xml
2022-11-23T02:32:25.8691418Z 
2022-11-23T02:32:25.8692115Z ##[endgroup]
2022-11-23T02:32:25.8692608Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_rtijl0w0)
2022-11-23T02:32:25.8692634Z 
2022-11-23T02:32:25.8692889Z Running distributed/test_c10d_nccl ... [2022-11-23 02:32:25.775112]
2022-11-23T02:32:25.8693366Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:32:25.775484]
2022-11-23T02:47:49.4221040Z 
2022-11-23T02:47:49.4224714Z Expand the folded group to see the log file of distributed/test_c10d_nccl
2022-11-23T02:47:49.4225969Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_b3knhzl_)
2022-11-23T02:47:49.4229066Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-11-23T02:47:49.4232627Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2022-11-23T02:47:49.4233245Z test_broadcast_coalesced_nccl (__main__.CommTest)
2022-11-23T02:47:49.4233803Z test_nccl_barrier (__main__.CommTest)
2022-11-23T02:47:49.4234279Z test_nccl_barrier_device_ids (__main__.CommTest)
2022-11-23T02:47:49.4234813Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2022-11-23T02:47:49.4235477Z test_nccl_barrier_timeout (__main__.CommTest)
2022-11-23T02:47:49.4235875Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2022-11-23T02:47:49.4236490Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2022-11-23T02:47:49.4237180Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2022-11-23T02:47:49.4237827Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2022-11-23T02:47:49.4238330Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2022-11-23T02:47:49.4238917Z test_nncl_rank_membership (__main__.CommTest)
2022-11-23T02:47:49.4239426Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2022-11-23T02:47:49.4240004Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2022-11-23T02:47:49.4240441Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2022-11-23T02:47:49.4241105Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2022-11-23T02:47:49.4241672Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2022-11-23T02:47:49.4242011Z test_tensor_dtype_complex (__main__.CommTest)
2022-11-23T02:47:49.4242377Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-11-23T02:47:49.4243858Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-11-23T02:47:49.4245089Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:47:49.4245509Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:47:49.4246011Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:47:49.4246506Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:47:49.4246898Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-11-23T02:47:49.4247459Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:47:49.4248043Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:47:49.4260694Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_packed_sequence>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-11-23T02:47:49.4274153Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4274655Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4275128Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4275604Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4276062Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4276497Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4277462Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4278181Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4278630Z test_channels_last_contig (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4279074Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4279557Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4280021Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4280510Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4281017Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4281551Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4282044Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4282556Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4283497Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4284177Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4284689Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4285205Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4285723Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4286280Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4286762Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4287247Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4287730Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4288193Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4288654Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4289082Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4289483Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4289917Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4290353Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4290803Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4291218Z test_failure_recovery (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4292123Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4292601Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4293065Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4293648Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4294178Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4294686Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4295105Z test_fp16 (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4295524Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4295963Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4296375Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4296828Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4297297Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4297733Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4298171Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4298651Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4299150Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4299635Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4300111Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4300544Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4301001Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4301467Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4301963Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4302461Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4302908Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4303327Z test_no_grad (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4303733Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4304236Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4304641Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4305101Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4327171Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4327627Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4328198Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:47:49.4329645Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2022-11-23T02:47:49.4330901Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2022-11-23T02:47:49.4331322Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2022-11-23T02:47:49.4332119Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2022-11-23T02:47:49.4332669Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2022-11-23T02:47:49.4333112Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2022-11-23T02:47:49.4333529Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2022-11-23T02:47:49.4333915Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2022-11-23T02:47:49.4334323Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2022-11-23T02:47:49.4334703Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2022-11-23T02:47:49.4335722Z <unittest.suite.TestSuite tests=[<__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allgather_base>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_reduce_scatter_base>]>
2022-11-23T02:47:49.4337132Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:47:49.4337694Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:47:49.4338232Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:47:49.4338768Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:47:49.4339538Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2022-11-23T02:47:49.4340024Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2022-11-23T02:47:49.4342065Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_nccl_dist_backend_error>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]>
2022-11-23T02:47:49.4344268Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4344645Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4345022Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4345391Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4345744Z test_barrier (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4346079Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4346446Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4346807Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4347146Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4347502Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4347878Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4348230Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4348613Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4349007Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4349448Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4349813Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4350178Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4350533Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4350870Z test_send_recv (__main__.ProcessGroupNCCLTest)
2022-11-23T02:47:49.4351309Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2022-11-23T02:47:49.4351724Z test_common_errors (__main__.RendezvousEnvTest)
2022-11-23T02:47:49.4352035Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:47:49.4352461Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2022-11-23T02:47:49.4352895Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2022-11-23T02:47:49.4353598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4354050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4354632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4355111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4355346Z 
2022-11-23T02:47:49.4355444Z Running tests...
2022-11-23T02:47:49.4355854Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4356385Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4356866Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4357315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76781
2022-11-23T02:47:49.4357772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76782
2022-11-23T02:47:49.4358383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4358821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4359395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4359862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4360514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4360943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4361514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4361985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4362411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4362888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4363885Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:47:49.4364520Z   warnings.warn(
2022-11-23T02:47:49.4365394Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:47:49.4365995Z   warnings.warn(
2022-11-23T02:47:49.4366237Z ok (5.345s)
2022-11-23T02:47:49.4366386Z 
2022-11-23T02:47:49.4366721Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4367047Z Ran 1 test in 5.346s
2022-11-23T02:47:49.4367208Z 
2022-11-23T02:47:49.4367303Z OK
2022-11-23T02:47:49.4367436Z 
2022-11-23T02:47:49.4367559Z Generating XML reports...
2022-11-23T02:47:49.4368090Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023229.xml
2022-11-23T02:47:49.4368755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4369209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4369782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4370236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4370466Z 
2022-11-23T02:47:49.4370573Z Running tests...
2022-11-23T02:47:49.4370977Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4371849Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4372328Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4372796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76898
2022-11-23T02:47:49.4373247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76899
2022-11-23T02:47:49.4373850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4374300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4374874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4375344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4375905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4376346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4376911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4377359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4377893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4378367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4378719Z ok (5.430s)
2022-11-23T02:47:49.4378851Z 
2022-11-23T02:47:49.4379127Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4379457Z Ran 1 test in 5.431s
2022-11-23T02:47:49.4379617Z 
2022-11-23T02:47:49.4379713Z OK
2022-11-23T02:47:49.4379848Z 
2022-11-23T02:47:49.4379956Z Generating XML reports...
2022-11-23T02:47:49.4380504Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023237.xml
2022-11-23T02:47:49.4381161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4381608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4382165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4382633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4382864Z 
2022-11-23T02:47:49.4382975Z Running tests...
2022-11-23T02:47:49.4383361Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4383891Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4384352Z   test_nccl_barrier (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4384867Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77015
2022-11-23T02:47:49.4385317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77016
2022-11-23T02:47:49.4385925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4386381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4386940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4387416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4387998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4388444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4389000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4389465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4389904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4390377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4390758Z skip: Need at least 4 CUDA devices (3.948s)
2022-11-23T02:47:49.4390950Z 
2022-11-23T02:47:49.4391216Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4391543Z Ran 1 test in 3.948s
2022-11-23T02:47:49.4391703Z 
2022-11-23T02:47:49.4391794Z OK (skipped=1)
2022-11-23T02:47:49.4391948Z 
2022-11-23T02:47:49.4392073Z Generating XML reports...
2022-11-23T02:47:49.4392612Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023245.xml
2022-11-23T02:47:49.4393277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4393711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4394283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4394749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4395043Z 
2022-11-23T02:47:49.4395135Z Running tests...
2022-11-23T02:47:49.4395539Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4396069Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4396546Z   test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4396995Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77118
2022-11-23T02:47:49.4397454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77119
2022-11-23T02:47:49.4398057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4398486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4399060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4399528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4400099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4400522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4401086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4401552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4402025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4402538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4403027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4403509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4404151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4404845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4405244Z ok (4.843s)
2022-11-23T02:47:49.4405394Z 
2022-11-23T02:47:49.4405661Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4405969Z Ran 1 test in 4.844s
2022-11-23T02:47:49.4406134Z 
2022-11-23T02:47:49.4406227Z OK
2022-11-23T02:47:49.4406362Z 
2022-11-23T02:47:49.4406488Z Generating XML reports...
2022-11-23T02:47:49.4407013Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023251.xml
2022-11-23T02:47:49.4407670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4408125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4408698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4409153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4409387Z 
2022-11-23T02:47:49.4409496Z Running tests...
2022-11-23T02:47:49.4409895Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4410408Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4410908Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4411396Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77234
2022-11-23T02:47:49.4412258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77235
2022-11-23T02:47:49.4412952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4413402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4413974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4414427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4415002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4415450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4416018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4416465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4416905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4417402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4417868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4418348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4419006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4419758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4420151Z ok (3.887s)
2022-11-23T02:47:49.4420300Z 
2022-11-23T02:47:49.4420571Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4420900Z Ran 1 test in 3.887s
2022-11-23T02:47:49.4421062Z 
2022-11-23T02:47:49.4421162Z OK
2022-11-23T02:47:49.4421278Z 
2022-11-23T02:47:49.4421405Z Generating XML reports...
2022-11-23T02:47:49.4421943Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023258.xml
2022-11-23T02:47:49.4422603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4423036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4423608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4424084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4424312Z 
2022-11-23T02:47:49.4424421Z Running tests...
2022-11-23T02:47:49.4424806Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4425335Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4425806Z   test_nccl_barrier_timeout (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4426250Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77341
2022-11-23T02:47:49.4426701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77342
2022-11-23T02:47:49.4427308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4427754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4428316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4428844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4429422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4429848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4430521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4430986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4431426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4431887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4432277Z skip: Need at least 4 CUDA devices (3.936s)
2022-11-23T02:47:49.4432468Z 
2022-11-23T02:47:49.4432743Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4433055Z Ran 1 test in 3.936s
2022-11-23T02:47:49.4433214Z 
2022-11-23T02:47:49.4433321Z OK (skipped=1)
2022-11-23T02:47:49.4433478Z 
2022-11-23T02:47:49.4433603Z Generating XML reports...
2022-11-23T02:47:49.4434141Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023305.xml
2022-11-23T02:47:49.4434786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4435237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4435809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4436280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4436494Z 
2022-11-23T02:47:49.4436602Z Running tests...
2022-11-23T02:47:49.4437170Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4437723Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4438196Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4438665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77444
2022-11-23T02:47:49.4439121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77445
2022-11-23T02:47:49.4439728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4440160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4440734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4441203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4441767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4442213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4442778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4443247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4443667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4444137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4444534Z skip: Need at least 4 CUDA devices (3.834s)
2022-11-23T02:47:49.4444725Z 
2022-11-23T02:47:49.4444977Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4445305Z Ran 1 test in 3.834s
2022-11-23T02:47:49.4445466Z 
2022-11-23T02:47:49.4445578Z OK (skipped=1)
2022-11-23T02:47:49.4445733Z 
2022-11-23T02:47:49.4445858Z Generating XML reports...
2022-11-23T02:47:49.4446381Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023311.xml
2022-11-23T02:47:49.4447035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4447550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4448110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4448581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4448810Z 
2022-11-23T02:47:49.4448920Z Running tests...
2022-11-23T02:47:49.4449323Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4449839Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4450343Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4450825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77547
2022-11-23T02:47:49.4451257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77548
2022-11-23T02:47:49.4452287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4452737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4453308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4453762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4454336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4454869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4455461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4455908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4456345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4456825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4457201Z skip: Need at least 4 CUDA devices (3.934s)
2022-11-23T02:47:49.4457393Z 
2022-11-23T02:47:49.4457663Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4457988Z Ran 1 test in 3.934s
2022-11-23T02:47:49.4458147Z 
2022-11-23T02:47:49.4458255Z OK (skipped=1)
2022-11-23T02:47:49.4458393Z 
2022-11-23T02:47:49.4458518Z Generating XML reports...
2022-11-23T02:47:49.4459057Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023317.xml
2022-11-23T02:47:49.4459713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4460147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4460719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4461190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4461418Z 
2022-11-23T02:47:49.4461525Z Running tests...
2022-11-23T02:47:49.4461912Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4462436Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4462921Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4463384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77650
2022-11-23T02:47:49.4463835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77651
2022-11-23T02:47:49.4464442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4464894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4465537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4466006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4466585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4467014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4467588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4468056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4468495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4468952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4469439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4469939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4470599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4471274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4471858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:47:49.4472363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:47:49.4472996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4473680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4474081Z ok (4.933s)
2022-11-23T02:47:49.4474229Z 
2022-11-23T02:47:49.4474497Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4474816Z Ran 1 test in 4.933s
2022-11-23T02:47:49.4474976Z 
2022-11-23T02:47:49.4475070Z OK
2022-11-23T02:47:49.4475201Z 
2022-11-23T02:47:49.4475326Z Generating XML reports...
2022-11-23T02:47:49.4475847Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023323.xml
2022-11-23T02:47:49.4476513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4476963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4477539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4478001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4478232Z 
2022-11-23T02:47:49.4478342Z Running tests...
2022-11-23T02:47:49.4478745Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4479256Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4479742Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4480211Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77781
2022-11-23T02:47:49.4480662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77782
2022-11-23T02:47:49.4481250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4481697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4482270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4482811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4483374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4483819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4484389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4484841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4485279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4485770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4486253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4486719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4487378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4487910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:47:49.4488554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4489145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:47:49.4489806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4490482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4490862Z ok (4.936s)
2022-11-23T02:47:49.4491010Z 
2022-11-23T02:47:49.4491277Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4491994Z Ran 1 test in 4.936s
2022-11-23T02:47:49.4492158Z 
2022-11-23T02:47:49.4492253Z OK
2022-11-23T02:47:49.4492372Z 
2022-11-23T02:47:49.4492497Z Generating XML reports...
2022-11-23T02:47:49.4493044Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023330.xml
2022-11-23T02:47:49.4493709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4494142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4494721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4495191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4495423Z 
2022-11-23T02:47:49.4495533Z Running tests...
2022-11-23T02:47:49.4495918Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4496445Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4496930Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4497383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77903
2022-11-23T02:47:49.4497835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77904
2022-11-23T02:47:49.4498438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4498891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4499449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4500018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4500606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4501051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4501601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4502064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4502507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4502983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4503469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4503951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4504605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4505122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:47:49.4505769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4506298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:47:49.4507010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4507696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4508092Z ok (4.944s)
2022-11-23T02:47:49.4508240Z 
2022-11-23T02:47:49.4508506Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4508825Z Ran 1 test in 4.944s
2022-11-23T02:47:49.4508985Z 
2022-11-23T02:47:49.4509079Z OK
2022-11-23T02:47:49.4509212Z 
2022-11-23T02:47:49.4509336Z Generating XML reports...
2022-11-23T02:47:49.4509876Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023338.xml
2022-11-23T02:47:49.4510526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4510978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4511553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4512008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4512238Z 
2022-11-23T02:47:49.4512347Z Running tests...
2022-11-23T02:47:49.4512748Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4513276Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4513728Z   test_nncl_rank_membership (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4514188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78025
2022-11-23T02:47:49.4514641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78026
2022-11-23T02:47:49.4515227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4515679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4516253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4516721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4517282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4517791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4518360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4518806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4519246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4519743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4520224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4520690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4521345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4521878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:47:49.4522524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4523098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:47:49.4523749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4524480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4524888Z ok (3.938s)
2022-11-23T02:47:49.4525019Z 
2022-11-23T02:47:49.4525290Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4525621Z Ran 1 test in 3.938s
2022-11-23T02:47:49.4525790Z 
2022-11-23T02:47:49.4525885Z OK
2022-11-23T02:47:49.4526021Z 
2022-11-23T02:47:49.4526129Z Generating XML reports...
2022-11-23T02:47:49.4526671Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023345.xml
2022-11-23T02:47:49.4527332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4527785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4528346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4528864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4529094Z 
2022-11-23T02:47:49.4529203Z Running tests...
2022-11-23T02:47:49.4529593Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4530121Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4530626Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4531111Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78134
2022-11-23T02:47:49.4531784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78135
2022-11-23T02:47:49.4532400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4532850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4533414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4533882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4534461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4535001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4535568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4536035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4536477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4536953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4537447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4537932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4538589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4539261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4539799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:47:49.4540293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:47:49.4540943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4541670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4542078Z ok (5.347s)
2022-11-23T02:47:49.4542228Z 
2022-11-23T02:47:49.4542498Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4542810Z Ran 1 test in 5.347s
2022-11-23T02:47:49.4542973Z 
2022-11-23T02:47:49.4543070Z OK
2022-11-23T02:47:49.4543206Z 
2022-11-23T02:47:49.4543329Z Generating XML reports...
2022-11-23T02:47:49.4543870Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023351.xml
2022-11-23T02:47:49.4544516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4544967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4545545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4546003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4546235Z 
2022-11-23T02:47:49.4546344Z Running tests...
2022-11-23T02:47:49.4546746Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4547277Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4547756Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4548243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78255
2022-11-23T02:47:49.4548692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78256
2022-11-23T02:47:49.4549281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4549730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4550307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4550781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4551337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4551781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4552416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4552882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4553303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4553777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4554267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4554755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4555414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4556104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4556643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:47:49.4557122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:47:49.4557765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4558447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4558843Z ok (4.893s)
2022-11-23T02:47:49.4558974Z 
2022-11-23T02:47:49.4559297Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4559636Z Ran 1 test in 4.893s
2022-11-23T02:47:49.4559799Z 
2022-11-23T02:47:49.4559894Z OK
2022-11-23T02:47:49.4560034Z 
2022-11-23T02:47:49.4560142Z Generating XML reports...
2022-11-23T02:47:49.4560684Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023359.xml
2022-11-23T02:47:49.4561353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4561803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4562360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4562832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4563062Z 
2022-11-23T02:47:49.4563171Z Running tests...
2022-11-23T02:47:49.4563559Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4564089Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4564589Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4565072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78377
2022-11-23T02:47:49.4565512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78378
2022-11-23T02:47:49.4566109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4566554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4567110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4567586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4568160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4568601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4569156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4569689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4570126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4570601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4570978Z skip: Need at least 4 CUDA devices (3.928s)
2022-11-23T02:47:49.4571172Z 
2022-11-23T02:47:49.4571664Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4572006Z Ran 1 test in 3.928s
2022-11-23T02:47:49.4572167Z 
2022-11-23T02:47:49.4572263Z OK (skipped=1)
2022-11-23T02:47:49.4572419Z 
2022-11-23T02:47:49.4572547Z Generating XML reports...
2022-11-23T02:47:49.4573090Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023406.xml
2022-11-23T02:47:49.4573751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4574190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4574766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4575232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4575461Z 
2022-11-23T02:47:49.4575552Z Running tests...
2022-11-23T02:47:49.4575951Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4576565Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4577072Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4577530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78480
2022-11-23T02:47:49.4577984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78481
2022-11-23T02:47:49.4578596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4579028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4579602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4580070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4580647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4581075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4581642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4582105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4582527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4583024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4583515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4584006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4584646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4585337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4585732Z ok (4.901s)
2022-11-23T02:47:49.4585880Z 
2022-11-23T02:47:49.4586151Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4586461Z Ran 1 test in 4.901s
2022-11-23T02:47:49.4586621Z 
2022-11-23T02:47:49.4586793Z OK
2022-11-23T02:47:49.4586932Z 
2022-11-23T02:47:49.4587059Z Generating XML reports...
2022-11-23T02:47:49.4587585Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023413.xml
2022-11-23T02:47:49.4588241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4588690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4589263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4589726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4589956Z 
2022-11-23T02:47:49.4590066Z Running tests...
2022-11-23T02:47:49.4590468Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4590980Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4591472Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4591948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78596
2022-11-23T02:47:49.4592401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78597
2022-11-23T02:47:49.4592995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4593448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4594071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4594534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4595112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4595557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4596127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4596571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4597014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4597508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4597996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4598458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4599117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4599647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:47:49.4600276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4600806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:47:49.4601448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4602125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:47:49.4602506Z ok (4.861s)
2022-11-23T02:47:49.4602654Z 
2022-11-23T02:47:49.4602922Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4603250Z Ran 1 test in 4.861s
2022-11-23T02:47:49.4603413Z 
2022-11-23T02:47:49.4603489Z OK
2022-11-23T02:47:49.4603622Z 
2022-11-23T02:47:49.4603749Z Generating XML reports...
2022-11-23T02:47:49.4604352Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023420.xml
2022-11-23T02:47:49.4605011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4605446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4606022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4606493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4606724Z 
2022-11-23T02:47:49.4606836Z Running tests...
2022-11-23T02:47:49.4607223Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4607751Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4608222Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4608668Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78716
2022-11-23T02:47:49.4609120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78717
2022-11-23T02:47:49.4609729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4610175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4610736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4611268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4612047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4612475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4613043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4613508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4613944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4614418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4614898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4615385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4616019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4616710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4617101Z ok (5.435s)
2022-11-23T02:47:49.4617253Z 
2022-11-23T02:47:49.4617526Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4617838Z Ran 1 test in 5.435s
2022-11-23T02:47:49.4617998Z 
2022-11-23T02:47:49.4618093Z OK
2022-11-23T02:47:49.4618227Z 
2022-11-23T02:47:49.4618351Z Generating XML reports...
2022-11-23T02:47:49.4618876Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023427.xml
2022-11-23T02:47:49.4619532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4619983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4620553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4621005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4621233Z 
2022-11-23T02:47:49.4621342Z Running tests...
2022-11-23T02:47:49.4621844Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4622376Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4622832Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4623294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78833
2022-11-23T02:47:49.4623747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78834
2022-11-23T02:47:49.4624341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4624794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4625368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4625833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4626394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4626837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4627402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4627846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4628281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4628902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4629398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4629868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4630530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4631224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4632267Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:47:49.4632886Z   warnings.warn(
2022-11-23T02:47:49.4633769Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:47:49.4634391Z   warnings.warn(
2022-11-23T02:47:49.4635257Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:47:49.4635857Z   warnings.warn(
2022-11-23T02:47:49.4636721Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:47:49.4637330Z   warnings.warn(
2022-11-23T02:47:49.4637568Z ok (4.797s)
2022-11-23T02:47:49.4637717Z 
2022-11-23T02:47:49.4637971Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4638301Z Ran 1 test in 4.797s
2022-11-23T02:47:49.4638462Z 
2022-11-23T02:47:49.4638557Z OK
2022-11-23T02:47:49.4638689Z 
2022-11-23T02:47:49.4638857Z Generating XML reports...
2022-11-23T02:47:49.4639406Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023435.xml
2022-11-23T02:47:49.4640068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4640516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4641077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4641552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4641781Z 
2022-11-23T02:47:49.4641888Z Running tests...
2022-11-23T02:47:49.4642274Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4642804Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4643291Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4643766Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78942
2022-11-23T02:47:49.4644204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78943
2022-11-23T02:47:49.4644799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4645249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4645873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4646343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4646922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4647369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4647925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4648392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4648836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4649303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4649771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4650271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4650926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4651862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4652794Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4653518Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4654369Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4655079Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4655400Z ok (5.337s)
2022-11-23T02:47:49.4655547Z 
2022-11-23T02:47:49.4655818Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4656148Z Ran 1 test in 5.337s
2022-11-23T02:47:49.4656413Z 
2022-11-23T02:47:49.4656491Z OK
2022-11-23T02:47:49.4656627Z 
2022-11-23T02:47:49.4656750Z Generating XML reports...
2022-11-23T02:47:49.4657312Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023442.xml
2022-11-23T02:47:49.4657990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4658425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4659000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4659469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4659699Z 
2022-11-23T02:47:49.4659807Z Running tests...
2022-11-23T02:47:49.4660191Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4660717Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4661204Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4661660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79057
2022-11-23T02:47:49.4662112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79058
2022-11-23T02:47:49.4662714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4663145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4663791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4664278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4664864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4665293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4665866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4666329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4666766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4667224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4667710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4668214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4668855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4669550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4670471Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4671188Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4672041Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4672737Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4673067Z ok (5.349s)
2022-11-23T02:47:49.4673214Z 
2022-11-23T02:47:49.4673542Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4673856Z Ran 1 test in 5.349s
2022-11-23T02:47:49.4674017Z 
2022-11-23T02:47:49.4674113Z OK
2022-11-23T02:47:49.4674246Z 
2022-11-23T02:47:49.4674371Z Generating XML reports...
2022-11-23T02:47:49.4674926Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023450.xml
2022-11-23T02:47:49.4675586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4676042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4676614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4677068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4677299Z 
2022-11-23T02:47:49.4677408Z Running tests...
2022-11-23T02:47:49.4677809Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4678346Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4678815Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4679285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79172
2022-11-23T02:47:49.4679738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79173
2022-11-23T02:47:49.4680382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4680847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4681424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4681896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4682460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4682904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4683471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4683935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4684356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4684828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4685317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4685801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4686458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4687148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4688069Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4688772Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4689627Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4690341Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4690729Z ok (5.438s)
2022-11-23T02:47:49.4690878Z 
2022-11-23T02:47:49.4691132Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4691627Z Ran 1 test in 5.438s
2022-11-23T02:47:49.4691799Z 
2022-11-23T02:47:49.4691895Z OK
2022-11-23T02:47:49.4692029Z 
2022-11-23T02:47:49.4692154Z Generating XML reports...
2022-11-23T02:47:49.4692700Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023457.xml
2022-11-23T02:47:49.4693383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4693835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4694397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4694871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4695107Z 
2022-11-23T02:47:49.4695217Z Running tests...
2022-11-23T02:47:49.4695618Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4696128Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4696622Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4697103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79287
2022-11-23T02:47:49.4697538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79288
2022-11-23T02:47:49.4698272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4698740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4699320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4699780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4700358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4700799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4701351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4701814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4702257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4702728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4703201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4703698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4704357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4705044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4705950Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4706669Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4707522Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4708317Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4709149Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4709859Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4710711Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4711409Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4712255Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4712947Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4713788Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4714546Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4715401Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4716085Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4716926Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4717628Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4717952Z ok (5.301s)
2022-11-23T02:47:49.4718103Z 
2022-11-23T02:47:49.4718356Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4718688Z Ran 1 test in 5.301s
2022-11-23T02:47:49.4718849Z 
2022-11-23T02:47:49.4718947Z OK
2022-11-23T02:47:49.4719080Z 
2022-11-23T02:47:49.4719204Z Generating XML reports...
2022-11-23T02:47:49.4719746Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023505.xml
2022-11-23T02:47:49.4720423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4720878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4721436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4721910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4722141Z 
2022-11-23T02:47:49.4722249Z Running tests...
2022-11-23T02:47:49.4722651Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4723163Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4723655Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4724130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79402
2022-11-23T02:47:49.4724564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79403
2022-11-23T02:47:49.4725241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4725694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4726266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4726717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4727298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4727741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4728308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4728804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4729253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4729723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4730197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4730694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4731632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4732360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4733262Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4733978Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4734835Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4735547Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4736398Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4737100Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4737944Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4738650Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4738978Z ok (5.405s)
2022-11-23T02:47:49.4739109Z 
2022-11-23T02:47:49.4739378Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4739706Z Ran 1 test in 5.405s
2022-11-23T02:47:49.4739867Z 
2022-11-23T02:47:49.4739961Z OK
2022-11-23T02:47:49.4740100Z 
2022-11-23T02:47:49.4740208Z Generating XML reports...
2022-11-23T02:47:49.4740761Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023513.xml
2022-11-23T02:47:49.4741435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4741885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4742538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4743011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4743244Z 
2022-11-23T02:47:49.4743355Z Running tests...
2022-11-23T02:47:49.4743739Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4744266Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4744761Z   test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4745238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79517
2022-11-23T02:47:49.4745672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79518
2022-11-23T02:47:49.4746269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4746722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4747277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4747749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4748323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4748832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4749406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4749874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4750316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4750780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4751270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4751772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4752427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4753100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4754024Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4754736Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4755592Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4756306Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4757145Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4757855Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4758698Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4759472Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4759781Z ok (5.301s)
2022-11-23T02:47:49.4759932Z 
2022-11-23T02:47:49.4760197Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4760528Z Ran 1 test in 5.301s
2022-11-23T02:47:49.4760688Z 
2022-11-23T02:47:49.4760766Z OK
2022-11-23T02:47:49.4760902Z 
2022-11-23T02:47:49.4761024Z Generating XML reports...
2022-11-23T02:47:49.4761580Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023520.xml
2022-11-23T02:47:49.4762254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4762686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4763258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4763730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4763959Z 
2022-11-23T02:47:49.4764067Z Running tests...
2022-11-23T02:47:49.4764452Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4764978Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4765456Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4765960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79632
2022-11-23T02:47:49.4766426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79633
2022-11-23T02:47:49.4767034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4767482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4768045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4768511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4769086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4769514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4770087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4770549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4770984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4771606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4772112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.4772616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.4773265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4773956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.4774881Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4775600Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4776456Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:47:49.4777244Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:47:49.4777575Z ok (5.432s)
2022-11-23T02:47:49.4777724Z 
2022-11-23T02:47:49.4777993Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4778306Z Ran 1 test in 5.432s
2022-11-23T02:47:49.4778468Z 
2022-11-23T02:47:49.4778561Z OK
2022-11-23T02:47:49.4778698Z 
2022-11-23T02:47:49.4778828Z Generating XML reports...
2022-11-23T02:47:49.4779383Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023528.xml
2022-11-23T02:47:49.4780040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4780489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4781068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4781521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4781749Z 
2022-11-23T02:47:49.4781859Z Running tests...
2022-11-23T02:47:49.4782259Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4782789Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4783371Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4783898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79749
2022-11-23T02:47:49.4784354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79750
2022-11-23T02:47:49.4784963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4785400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4785977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4786446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4787008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4787460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4788030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4788493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4788913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4789389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4789893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5z3y3ld
2022-11-23T02:47:49.4790415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5z3y3ld/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4790947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8p9nw46
2022-11-23T02:47:49.4791483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8p9nw46/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4791999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4792470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4792822Z ok (5.831s)
2022-11-23T02:47:49.4792970Z 
2022-11-23T02:47:49.4793239Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4793611Z Ran 1 test in 5.831s
2022-11-23T02:47:49.4793774Z 
2022-11-23T02:47:49.4793868Z OK
2022-11-23T02:47:49.4794002Z 
2022-11-23T02:47:49.4794125Z Generating XML reports...
2022-11-23T02:47:49.4794748Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023536.xml
2022-11-23T02:47:49.4795455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4795907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4796485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4796938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4797170Z 
2022-11-23T02:47:49.4797279Z Running tests...
2022-11-23T02:47:49.4797681Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4798216Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4798753Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4799284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79870
2022-11-23T02:47:49.4799737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79871
2022-11-23T02:47:49.4800394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4800838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4801402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4801851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4802231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4802411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4802787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4802977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4803210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4803433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4803689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy4kb56k
2022-11-23T02:47:49.4803959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy4kb56k/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4804210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yelfo3m
2022-11-23T02:47:49.4804484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yelfo3m/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4804704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4804938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4805040Z ok (5.802s)
2022-11-23T02:47:49.4805061Z 
2022-11-23T02:47:49.4805331Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4805444Z Ran 1 test in 5.803s
2022-11-23T02:47:49.4805467Z 
2022-11-23T02:47:49.4805566Z OK
2022-11-23T02:47:49.4805583Z 
2022-11-23T02:47:49.4805707Z Generating XML reports...
2022-11-23T02:47:49.4806171Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023544.xml
2022-11-23T02:47:49.4806526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4806761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4807143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4807336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4807356Z 
2022-11-23T02:47:49.4807467Z Running tests...
2022-11-23T02:47:49.4807728Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4808043Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4808330Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4808532Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79991
2022-11-23T02:47:49.4808752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79992
2022-11-23T02:47:49.4809129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4809303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4809683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4809874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4810287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4810470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4810843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4811015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4811250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4811701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4811969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9tcmozcw
2022-11-23T02:47:49.4812243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9tcmozcw/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4812501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqitpw159
2022-11-23T02:47:49.4812771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqitpw159/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4813004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4813241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4813325Z ok (5.834s)
2022-11-23T02:47:49.4813349Z 
2022-11-23T02:47:49.4813626Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4813739Z Ran 1 test in 5.834s
2022-11-23T02:47:49.4813759Z 
2022-11-23T02:47:49.4813853Z OK
2022-11-23T02:47:49.4813873Z 
2022-11-23T02:47:49.4813997Z Generating XML reports...
2022-11-23T02:47:49.4814460Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023552.xml
2022-11-23T02:47:49.4814832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4815011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4815375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4815566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4815586Z 
2022-11-23T02:47:49.4815695Z Running tests...
2022-11-23T02:47:49.4816062Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4816376Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4816679Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4816904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80112
2022-11-23T02:47:49.4817125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80113
2022-11-23T02:47:49.4817482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4817657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4818035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4818226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4818593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4818765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4819140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4819331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4819623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4819843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4820103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp652esiul
2022-11-23T02:47:49.4820377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp652esiul/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4820637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6qori8g
2022-11-23T02:47:49.4820903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6qori8g/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4821140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4821378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4821481Z ok (5.811s)
2022-11-23T02:47:49.4821501Z 
2022-11-23T02:47:49.4823796Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4823892Z Ran 1 test in 5.812s
2022-11-23T02:47:49.4823912Z 
2022-11-23T02:47:49.4824007Z OK
2022-11-23T02:47:49.4824027Z 
2022-11-23T02:47:49.4824151Z Generating XML reports...
2022-11-23T02:47:49.4824617Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023600.xml
2022-11-23T02:47:49.4824993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4825168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4825548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4825739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4825759Z 
2022-11-23T02:47:49.4825867Z Running tests...
2022-11-23T02:47:49.4826115Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4826426Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4826710Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4826929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80233
2022-11-23T02:47:49.4827214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80234
2022-11-23T02:47:49.4827586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4827762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4828138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4828313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4828722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4828895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4829268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4829465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4829693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4830242Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.4830535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4831079Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.4831343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp9qs5k44
2022-11-23T02:47:49.4831595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp9qs5k44/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4831849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzb8jfdu
2022-11-23T02:47:49.4832115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzb8jfdu/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4832218Z ok (5.295s)
2022-11-23T02:47:49.4832238Z 
2022-11-23T02:47:49.4832512Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4832623Z Ran 1 test in 5.295s
2022-11-23T02:47:49.4832643Z 
2022-11-23T02:47:49.4832737Z OK
2022-11-23T02:47:49.4832756Z 
2022-11-23T02:47:49.4832881Z Generating XML reports...
2022-11-23T02:47:49.4833345Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023608.xml
2022-11-23T02:47:49.4833705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4833881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4834263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4834455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4834475Z 
2022-11-23T02:47:49.4834584Z Running tests...
2022-11-23T02:47:49.4834847Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4835158Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4835440Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4835645Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80354
2022-11-23T02:47:49.4835920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80355
2022-11-23T02:47:49.4836293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4836469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4836848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4837041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4837405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4837579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4837951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4838126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4838355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4838899Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.4839170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4839718Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.4839978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalkabxie
2022-11-23T02:47:49.4840249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalkabxie/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4840500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47canely
2022-11-23T02:47:49.4840768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47canely/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4840870Z ok (5.319s)
2022-11-23T02:47:49.4840894Z 
2022-11-23T02:47:49.4841168Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4841265Z Ran 1 test in 5.319s
2022-11-23T02:47:49.4841285Z 
2022-11-23T02:47:49.4841380Z OK
2022-11-23T02:47:49.4841400Z 
2022-11-23T02:47:49.4841523Z Generating XML reports...
2022-11-23T02:47:49.4841986Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023616.xml
2022-11-23T02:47:49.4842359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4842535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4842914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4843104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4843124Z 
2022-11-23T02:47:49.4843219Z Running tests...
2022-11-23T02:47:49.4843484Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4843795Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4844074Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4844348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80475
2022-11-23T02:47:49.4844566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80476
2022-11-23T02:47:49.4844937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4845114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4845492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4845669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4846035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4846209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4846586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4846780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4847007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4847238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4847493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnhox0tz
2022-11-23T02:47:49.4847780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjf8x7lly
2022-11-23T02:47:49.4848061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnhox0tz/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4848321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjf8x7lly/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4848424Z ok (5.445s)
2022-11-23T02:47:49.4848445Z 
2022-11-23T02:47:49.4848711Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4848827Z Ran 1 test in 5.445s
2022-11-23T02:47:49.4848847Z 
2022-11-23T02:47:49.4848941Z OK
2022-11-23T02:47:49.4848960Z 
2022-11-23T02:47:49.4849083Z Generating XML reports...
2022-11-23T02:47:49.4849527Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023624.xml
2022-11-23T02:47:49.4849897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4850077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4850452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4850643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4850663Z 
2022-11-23T02:47:49.4850771Z Running tests...
2022-11-23T02:47:49.4851031Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4851347Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4851821Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4852033Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80596
2022-11-23T02:47:49.4852251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80597
2022-11-23T02:47:49.4852634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4852811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4853190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4853381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4853837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4854012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4854389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4854561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4854793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4855017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4855274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwcdbngq
2022-11-23T02:47:49.4855544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwcdbngq/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4855798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpft1atpj3
2022-11-23T02:47:49.4856069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpft1atpj3/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4856171Z ok (5.342s)
2022-11-23T02:47:49.4856191Z 
2022-11-23T02:47:49.4856437Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4856548Z Ran 1 test in 5.342s
2022-11-23T02:47:49.4856567Z 
2022-11-23T02:47:49.4856660Z OK
2022-11-23T02:47:49.4856679Z 
2022-11-23T02:47:49.4856804Z Generating XML reports...
2022-11-23T02:47:49.4857331Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023632.xml
2022-11-23T02:47:49.4857714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4857890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4858269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4858464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4858484Z 
2022-11-23T02:47:49.4858576Z Running tests...
2022-11-23T02:47:49.4858839Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4859152Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4859424Z   test_channels_last_contig (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4859646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80717
2022-11-23T02:47:49.4859864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80718
2022-11-23T02:47:49.4860231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4860410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4860775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4860966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4861329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4861500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4861881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4862070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4862300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4862528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4862670Z ok (5.406s)
2022-11-23T02:47:49.4862707Z 
2022-11-23T02:47:49.4862958Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4863069Z Ran 1 test in 5.407s
2022-11-23T02:47:49.4863088Z 
2022-11-23T02:47:49.4863183Z OK
2022-11-23T02:47:49.4863203Z 
2022-11-23T02:47:49.4863327Z Generating XML reports...
2022-11-23T02:47:49.4863783Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023639.xml
2022-11-23T02:47:49.4864154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4864328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4864703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4864873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4864895Z 
2022-11-23T02:47:49.4865004Z Running tests...
2022-11-23T02:47:49.4865264Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4865576Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4865805Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4866173Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4866438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80834
2022-11-23T02:47:49.4866663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80835
2022-11-23T02:47:49.4867018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4867193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4867570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4867760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4868124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4868298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4868666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4868858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4869088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4869299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4869553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6uv1ve5f
2022-11-23T02:47:49.4869827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6uv1ve5f/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4870083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zng4m3p
2022-11-23T02:47:49.4870350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zng4m3p/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4870452Z ok (5.426s)
2022-11-23T02:47:49.4870472Z 
2022-11-23T02:47:49.4870739Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4870856Z Ran 1 test in 5.427s
2022-11-23T02:47:49.4870875Z 
2022-11-23T02:47:49.4870952Z OK
2022-11-23T02:47:49.4870988Z 
2022-11-23T02:47:49.4871095Z Generating XML reports...
2022-11-23T02:47:49.4871556Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023647.xml
2022-11-23T02:47:49.4871927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4872159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4872540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4872731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4872750Z 
2022-11-23T02:47:49.4872859Z Running tests...
2022-11-23T02:47:49.4873122Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4873419Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4873661Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4873929Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4874148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80954
2022-11-23T02:47:49.4874368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80955
2022-11-23T02:47:49.4874740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4874916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4875298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4875519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4875891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4876063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4876432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4876625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4876856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4877087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4877345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvtbs8hi
2022-11-23T02:47:49.4877615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvtbs8hi/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4877856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xdrke50
2022-11-23T02:47:49.4878124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xdrke50/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4878226Z ok (5.334s)
2022-11-23T02:47:49.4878246Z 
2022-11-23T02:47:49.4878512Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4878627Z Ran 1 test in 5.334s
2022-11-23T02:47:49.4878647Z 
2022-11-23T02:47:49.4878740Z OK
2022-11-23T02:47:49.4878759Z 
2022-11-23T02:47:49.4878885Z Generating XML reports...
2022-11-23T02:47:49.4879350Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023655.xml
2022-11-23T02:47:49.4879703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4879878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4880258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4880450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4880469Z 
2022-11-23T02:47:49.4880580Z Running tests...
2022-11-23T02:47:49.4880839Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4881210Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4881451Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4881704Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4881907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81074
2022-11-23T02:47:49.4882123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81075
2022-11-23T02:47:49.4882497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4882674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4883051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4883241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4883607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4883781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4884140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4884331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4884608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4884843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4885099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr7lbms1
2022-11-23T02:47:49.4885372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr7lbms1/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4885628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j8ujjjj
2022-11-23T02:47:49.4885894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j8ujjjj/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4886134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4886353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4886581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4886816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4887728Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:47:49.4887847Z   warnings.warn(
2022-11-23T02:47:49.4888756Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:47:49.4888871Z   warnings.warn(
2022-11-23T02:47:49.4889103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4889329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4889557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4889772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4890108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4890336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4890558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4890783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4890884Z ok (5.532s)
2022-11-23T02:47:49.4890904Z 
2022-11-23T02:47:49.4891175Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4891289Z Ran 1 test in 5.532s
2022-11-23T02:47:49.4891309Z 
2022-11-23T02:47:49.4891385Z OK
2022-11-23T02:47:49.4891637Z 
2022-11-23T02:47:49.4891764Z Generating XML reports...
2022-11-23T02:47:49.4892246Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023702.xml
2022-11-23T02:47:49.4892623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4892800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4893179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4893370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4893390Z 
2022-11-23T02:47:49.4893498Z Running tests...
2022-11-23T02:47:49.4893856Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4894164Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4894401Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4894651Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4894873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81194
2022-11-23T02:47:49.4895093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81195
2022-11-23T02:47:49.4895465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4895641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4896024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4896199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4896570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4896744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4897119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4897305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4897536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4897766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4898023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1ucg9lf
2022-11-23T02:47:49.4898296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1ucg9lf/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4898531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36wvykb3
2022-11-23T02:47:49.4898798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36wvykb3/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4899032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4899339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4899570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4899804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4900722Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:47:49.4900836Z   warnings.warn(
2022-11-23T02:47:49.4901745Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:47:49.4901860Z   warnings.warn(
2022-11-23T02:47:49.4902076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4902302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4902578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4902817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4903047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4903272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4903497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4903723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4903806Z ok (5.404s)
2022-11-23T02:47:49.4903843Z 
2022-11-23T02:47:49.4904096Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4904209Z Ran 1 test in 5.404s
2022-11-23T02:47:49.4904228Z 
2022-11-23T02:47:49.4904323Z OK
2022-11-23T02:47:49.4904342Z 
2022-11-23T02:47:49.4904467Z Generating XML reports...
2022-11-23T02:47:49.4904933Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023710.xml
2022-11-23T02:47:49.4905306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4905483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4905866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4906040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4906060Z 
2022-11-23T02:47:49.4906168Z Running tests...
2022-11-23T02:47:49.4906431Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4906745Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4907014Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4907364Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4907583Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81314
2022-11-23T02:47:49.4907803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81315
2022-11-23T02:47:49.4908216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4908394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4908771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4908961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4909327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4909503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4909876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4910066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4910295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4910512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4910771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq85rgfo
2022-11-23T02:47:49.4911043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq85rgfo/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4911299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsm9grg4t
2022-11-23T02:47:49.4911617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsm9grg4t/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4911857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4912087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4912320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4912543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4912647Z ok (5.327s)
2022-11-23T02:47:49.4912667Z 
2022-11-23T02:47:49.4912936Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4913049Z Ran 1 test in 5.328s
2022-11-23T02:47:49.4913069Z 
2022-11-23T02:47:49.4913162Z OK
2022-11-23T02:47:49.4913181Z 
2022-11-23T02:47:49.4913304Z Generating XML reports...
2022-11-23T02:47:49.4913768Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023718.xml
2022-11-23T02:47:49.4914138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4914314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4914677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4914872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4914891Z 
2022-11-23T02:47:49.4914999Z Running tests...
2022-11-23T02:47:49.4915260Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4915575Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4915841Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4916193Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4916413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81434
2022-11-23T02:47:49.4916615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81435
2022-11-23T02:47:49.4916986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4917221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4917603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4917793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4918155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4918327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4918701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4918889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4919101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4919326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4919588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjo846tv_
2022-11-23T02:47:49.4919855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjo846tv_/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4920110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl265t46n
2022-11-23T02:47:49.4920375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl265t46n/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4920654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4920895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4921112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4921338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4921445Z ok (5.437s)
2022-11-23T02:47:49.4921464Z 
2022-11-23T02:47:49.4921734Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4921845Z Ran 1 test in 5.437s
2022-11-23T02:47:49.4921864Z 
2022-11-23T02:47:49.4921958Z OK
2022-11-23T02:47:49.4921977Z 
2022-11-23T02:47:49.4922103Z Generating XML reports...
2022-11-23T02:47:49.4922564Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023726.xml
2022-11-23T02:47:49.4922936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4923097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4923479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4923671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4923694Z 
2022-11-23T02:47:49.4923802Z Running tests...
2022-11-23T02:47:49.4924065Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4924379Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4924620Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4924993Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4925200Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81554
2022-11-23T02:47:49.4925418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81555
2022-11-23T02:47:49.4925790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4925966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4926401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4926593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4926956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4927130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4927503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4927681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4927911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4928139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4928394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31z9ggks
2022-11-23T02:47:49.4928716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31z9ggks/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4928973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8omjai1
2022-11-23T02:47:49.4929240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8omjai1/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4929476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4929745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4930533Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:47:49.4931332Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:47:49.4931793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4932020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4932127Z ok (5.436s)
2022-11-23T02:47:49.4932148Z 
2022-11-23T02:47:49.4932425Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4932538Z Ran 1 test in 5.436s
2022-11-23T02:47:49.4932558Z 
2022-11-23T02:47:49.4932651Z OK
2022-11-23T02:47:49.4932671Z 
2022-11-23T02:47:49.4932793Z Generating XML reports...
2022-11-23T02:47:49.4933255Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023734.xml
2022-11-23T02:47:49.4933631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4933792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4934172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4934362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4934382Z 
2022-11-23T02:47:49.4934575Z Running tests...
2022-11-23T02:47:49.4934840Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4935152Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4935392Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4935765Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4935985Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81674
2022-11-23T02:47:49.4936191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81675
2022-11-23T02:47:49.4936567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4936744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4937124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4937318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4937687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4937861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4938234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4938472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4938712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4938939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4939197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibb0sfv6
2022-11-23T02:47:49.4939472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibb0sfv6/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4939726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubr7nls4
2022-11-23T02:47:49.4939996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubr7nls4/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4940099Z ok (5.441s)
2022-11-23T02:47:49.4940118Z 
2022-11-23T02:47:49.4940386Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4940483Z Ran 1 test in 5.441s
2022-11-23T02:47:49.4940506Z 
2022-11-23T02:47:49.4940600Z OK
2022-11-23T02:47:49.4940619Z 
2022-11-23T02:47:49.4940743Z Generating XML reports...
2022-11-23T02:47:49.4941206Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023741.xml
2022-11-23T02:47:49.4941576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4941753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4942129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4942320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4942340Z 
2022-11-23T02:47:49.4942432Z Running tests...
2022-11-23T02:47:49.4942696Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4943011Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4943250Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4943523Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4943742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81794
2022-11-23T02:47:49.4944013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81795
2022-11-23T02:47:49.4944387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4944562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4944918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4945108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4945472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4945645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4946014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4946205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4946434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4946662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4946902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofllyi1u
2022-11-23T02:47:49.4947173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofllyi1u/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4947475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkcyl_hi
2022-11-23T02:47:49.4947750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkcyl_hi/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4947984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4948217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4948453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4948686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4948771Z ok (5.344s)
2022-11-23T02:47:49.4948808Z 
2022-11-23T02:47:49.4949062Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4949174Z Ran 1 test in 5.344s
2022-11-23T02:47:49.4949194Z 
2022-11-23T02:47:49.4949289Z OK
2022-11-23T02:47:49.4949308Z 
2022-11-23T02:47:49.4949434Z Generating XML reports...
2022-11-23T02:47:49.4949900Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023749.xml
2022-11-23T02:47:49.4950270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4950446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4950829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4951003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4951039Z 
2022-11-23T02:47:49.4951131Z Running tests...
2022-11-23T02:47:49.4951393Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4951702Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4951963Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4952235Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4952454Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81914
2022-11-23T02:47:49.4952671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81915
2022-11-23T02:47:49.4953104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4953265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4953645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4953836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4954203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4954376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4954754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4954942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4955175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4955385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4955641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8pxsxn7
2022-11-23T02:47:49.4955911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8pxsxn7/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4956166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzv9twop
2022-11-23T02:47:49.4956494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzv9twop/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4957279Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:47:49.4958061Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:47:49.4958974Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:47:49.4959094Z   warnings.warn(
2022-11-23T02:47:49.4960004Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:47:49.4960116Z   warnings.warn(
2022-11-23T02:47:49.4960353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4960587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4960817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4961089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4961193Z ok (5.446s)
2022-11-23T02:47:49.4961213Z 
2022-11-23T02:47:49.4961484Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4961597Z Ran 1 test in 5.446s
2022-11-23T02:47:49.4961617Z 
2022-11-23T02:47:49.4961711Z OK
2022-11-23T02:47:49.4961730Z 
2022-11-23T02:47:49.4961855Z Generating XML reports...
2022-11-23T02:47:49.4962323Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023757.xml
2022-11-23T02:47:49.4962695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4962855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4963233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4963430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4963449Z 
2022-11-23T02:47:49.4963557Z Running tests...
2022-11-23T02:47:49.4963824Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4964137Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4964393Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4964712Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4964939Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82034
2022-11-23T02:47:49.4965141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82035
2022-11-23T02:47:49.4965518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4965698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4966075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4966267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4966629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4966805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4967179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4967369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4967583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4967815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4968072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprya3m0a
2022-11-23T02:47:49.4968329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcalugup5
2022-11-23T02:47:49.4968598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprya3m0a/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4968870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcalugup5/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4969778Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:47:49.4969946Z   warnings.warn(
2022-11-23T02:47:49.4970852Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:47:49.4970964Z   warnings.warn(
2022-11-23T02:47:49.4971187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4971417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4971897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4972131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4972233Z ok (5.540s)
2022-11-23T02:47:49.4972257Z 
2022-11-23T02:47:49.4972531Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4972643Z Ran 1 test in 5.540s
2022-11-23T02:47:49.4972664Z 
2022-11-23T02:47:49.4972758Z OK
2022-11-23T02:47:49.4972777Z 
2022-11-23T02:47:49.4972885Z Generating XML reports...
2022-11-23T02:47:49.4973349Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023805.xml
2022-11-23T02:47:49.4973722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4973977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4974369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4974561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4974581Z 
2022-11-23T02:47:49.4974693Z Running tests...
2022-11-23T02:47:49.4974956Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4975265Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4975509Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4975747Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4975967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82154
2022-11-23T02:47:49.4976187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82155
2022-11-23T02:47:49.4976557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4976732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4977107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4977300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4977644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4977817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4978186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4978386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4978615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4978843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4979099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6f2ge63
2022-11-23T02:47:49.4979450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6f2ge63/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4979704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_usdjb3
2022-11-23T02:47:49.4979952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_usdjb3/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4980186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4980422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4980659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4980889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4980993Z ok (5.436s)
2022-11-23T02:47:49.4981013Z 
2022-11-23T02:47:49.4981284Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4981401Z Ran 1 test in 5.436s
2022-11-23T02:47:49.4981421Z 
2022-11-23T02:47:49.4981497Z OK
2022-11-23T02:47:49.4981516Z 
2022-11-23T02:47:49.4981640Z Generating XML reports...
2022-11-23T02:47:49.4982111Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023812.xml
2022-11-23T02:47:49.4982481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4982658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4983092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4983292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4983311Z 
2022-11-23T02:47:49.4983421Z Running tests...
2022-11-23T02:47:49.4983683Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4983984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4984247Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.4984485Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4984703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82274
2022-11-23T02:47:49.4984921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82275
2022-11-23T02:47:49.4985292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4985468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4985847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4986021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4986387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4986559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4986935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4987121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4987355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4987577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4987835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc09_ojri
2022-11-23T02:47:49.4988106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc09_ojri/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4988402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqoexl_x
2022-11-23T02:47:49.4988677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqoexl_x/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4988910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4989147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4989380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4989609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4989842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4990069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4990283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4990510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.4990613Z ok (5.433s)
2022-11-23T02:47:49.4990633Z 
2022-11-23T02:47:49.4990901Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4991013Z Ran 1 test in 5.433s
2022-11-23T02:47:49.4991032Z 
2022-11-23T02:47:49.4991126Z OK
2022-11-23T02:47:49.4991145Z 
2022-11-23T02:47:49.4991269Z Generating XML reports...
2022-11-23T02:47:49.4991777Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023820.xml
2022-11-23T02:47:49.4992143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4992323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4992701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4992897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4992917Z 
2022-11-23T02:47:49.4993026Z Running tests...
2022-11-23T02:47:49.4993285Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4993595Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.4993884Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.4994106Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82394
2022-11-23T02:47:49.4994310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82395
2022-11-23T02:47:49.4994678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4994853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4995233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4995422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4995787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4995959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.4996333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.4996507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.4996736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.4996964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.4997276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc5kkq69
2022-11-23T02:47:49.4997545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc5kkq69/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4997795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_nz7_oka
2022-11-23T02:47:49.4998060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_nz7_oka/_remote_module_non_scriptable.py
2022-11-23T02:47:49.4998163Z ok (5.336s)
2022-11-23T02:47:49.4998183Z 
2022-11-23T02:47:49.4998460Z ----------------------------------------------------------------------
2022-11-23T02:47:49.4998556Z Ran 1 test in 5.336s
2022-11-23T02:47:49.4998576Z 
2022-11-23T02:47:49.4998668Z OK
2022-11-23T02:47:49.4998687Z 
2022-11-23T02:47:49.4998810Z Generating XML reports...
2022-11-23T02:47:49.4999270Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023828.xml
2022-11-23T02:47:49.4999642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.4999818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5000193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5000384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5000404Z 
2022-11-23T02:47:49.5000510Z Running tests...
2022-11-23T02:47:49.5000802Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5001135Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5001441Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5001663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82515
2022-11-23T02:47:49.5001886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82516
2022-11-23T02:47:49.5002258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5002416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5002794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5002984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5003350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5003524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5003896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5004088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5004317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5004528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5004786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjs2cyr2
2022-11-23T02:47:49.5005056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjs2cyr2/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5005312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnwy1u0e
2022-11-23T02:47:49.5005578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnwy1u0e/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5005680Z ok (5.342s)
2022-11-23T02:47:49.5005700Z 
2022-11-23T02:47:49.5005965Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5006135Z Ran 1 test in 5.342s
2022-11-23T02:47:49.5006155Z 
2022-11-23T02:47:49.5006250Z OK
2022-11-23T02:47:49.5006269Z 
2022-11-23T02:47:49.5006375Z Generating XML reports...
2022-11-23T02:47:49.5006841Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023836.xml
2022-11-23T02:47:49.5007208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5007381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5007761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5007951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5007971Z 
2022-11-23T02:47:49.5008080Z Running tests...
2022-11-23T02:47:49.5008342Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5008636Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5008948Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5009167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82636
2022-11-23T02:47:49.5009385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82637
2022-11-23T02:47:49.5009754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5009976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5010365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5010555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5010920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5011080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5011666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5011868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5012100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5012332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5012593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdn15mjw_
2022-11-23T02:47:49.5012861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdn15mjw_/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5013112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q_07brx
2022-11-23T02:47:49.5013363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q_07brx/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5013465Z ok (5.461s)
2022-11-23T02:47:49.5013485Z 
2022-11-23T02:47:49.5013754Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5013865Z Ran 1 test in 5.461s
2022-11-23T02:47:49.5013885Z 
2022-11-23T02:47:49.5013978Z OK
2022-11-23T02:47:49.5013998Z 
2022-11-23T02:47:49.5014120Z Generating XML reports...
2022-11-23T02:47:49.5014585Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023843.xml
2022-11-23T02:47:49.5014956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5015133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5015495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5015774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5015794Z 
2022-11-23T02:47:49.5015906Z Running tests...
2022-11-23T02:47:49.5016178Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5016491Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5016731Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.5017024Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5017245Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82757
2022-11-23T02:47:49.5017446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82758
2022-11-23T02:47:49.5017813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5017991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5018364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5018553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5018912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5019082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5019527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5019729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5019943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5020172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5020434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpai0wka
2022-11-23T02:47:49.5020707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpai0wka/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5020957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20di0hh4
2022-11-23T02:47:49.5021221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20di0hh4/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5021325Z ok (5.297s)
2022-11-23T02:47:49.5021348Z 
2022-11-23T02:47:49.5021617Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5021712Z Ran 1 test in 5.297s
2022-11-23T02:47:49.5021750Z 
2022-11-23T02:47:49.5021826Z OK
2022-11-23T02:47:49.5021845Z 
2022-11-23T02:47:49.5021968Z Generating XML reports...
2022-11-23T02:47:49.5022430Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023851.xml
2022-11-23T02:47:49.5022800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5022974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5023351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5023542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5023561Z 
2022-11-23T02:47:49.5023672Z Running tests...
2022-11-23T02:47:49.5023915Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5024225Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5024454Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.5024746Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5025024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82878
2022-11-23T02:47:49.5025242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82879
2022-11-23T02:47:49.5025614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5025789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5026150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5026342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5026703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5026878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5027252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5027438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5027667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5027894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5028238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y9_3fe4
2022-11-23T02:47:49.5028498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y9_3fe4/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5028802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_naq0cs
2022-11-23T02:47:49.5029073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_naq0cs/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5029182Z ok (5.326s)
2022-11-23T02:47:49.5029202Z 
2022-11-23T02:47:49.5029470Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5029584Z Ran 1 test in 5.327s
2022-11-23T02:47:49.5029604Z 
2022-11-23T02:47:49.5029698Z OK
2022-11-23T02:47:49.5029718Z 
2022-11-23T02:47:49.5029843Z Generating XML reports...
2022-11-23T02:47:49.5030285Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023859.xml
2022-11-23T02:47:49.5030659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5030834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5031208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5031397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5031419Z 
2022-11-23T02:47:49.5031529Z Running tests...
2022-11-23T02:47:49.5031790Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5032103Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5032388Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5032590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82999
2022-11-23T02:47:49.5032810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83000
2022-11-23T02:47:49.5033178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5033353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5033732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5033982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5034355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5034529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5034884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5035072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5035308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5035539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5035688Z skip: Need at least 4 CUDA devices (3.908s)
2022-11-23T02:47:49.5035709Z 
2022-11-23T02:47:49.5035972Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5036086Z Ran 1 test in 3.908s
2022-11-23T02:47:49.5036106Z 
2022-11-23T02:47:49.5036215Z OK (skipped=1)
2022-11-23T02:47:49.5036235Z 
2022-11-23T02:47:49.5036358Z Generating XML reports...
2022-11-23T02:47:49.5036803Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023906.xml
2022-11-23T02:47:49.5037171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5037345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5037766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5037965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5037985Z 
2022-11-23T02:47:49.5038095Z Running tests...
2022-11-23T02:47:49.5038359Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5038675Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5038863Z   test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.5039122Z Tests that DDP with ``device_ids`` specified can run a forward and ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5039343Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83102
2022-11-23T02:47:49.5039563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83103
2022-11-23T02:47:49.5039938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5040113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5040491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5040684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5041047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5041203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5041575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5041761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5041995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5042241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5042465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5042711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5043175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5043571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5043813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6572tqw
2022-11-23T02:47:49.5044083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6572tqw/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5044340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jfy2e8k
2022-11-23T02:47:49.5044609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jfy2e8k/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5045555Z /opt/conda/lib/python3.10/site-packages/torch/distributed/_shard/replicated_tensor.py:113: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cudnn/RNN.cpp:982.)
2022-11-23T02:47:49.5045680Z   rs = func(*args, **kwargs)
2022-11-23T02:47:49.5046659Z /opt/conda/lib/python3.10/site-packages/torch/distributed/_shard/replicated_tensor.py:113: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cudnn/RNN.cpp:982.)
2022-11-23T02:47:49.5046786Z   rs = func(*args, **kwargs)
2022-11-23T02:47:49.5046889Z ok (6.148s)
2022-11-23T02:47:49.5046909Z 
2022-11-23T02:47:49.5047176Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5047270Z Ran 1 test in 6.148s
2022-11-23T02:47:49.5047309Z 
2022-11-23T02:47:49.5047386Z OK
2022-11-23T02:47:49.5047406Z 
2022-11-23T02:47:49.5047531Z Generating XML reports...
2022-11-23T02:47:49.5047992Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023913.xml
2022-11-23T02:47:49.5048361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5048537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5048917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5049108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5049128Z 
2022-11-23T02:47:49.5049236Z Running tests...
2022-11-23T02:47:49.5049479Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5049791Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5050067Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5050288Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83223
2022-11-23T02:47:49.5050507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83224
2022-11-23T02:47:49.5050877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5051056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5051661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5051852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5052223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5052487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5052864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5053054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5053287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5053517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5053779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcj9xmm7d
2022-11-23T02:47:49.5054050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcj9xmm7d/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5054289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4avvr5s
2022-11-23T02:47:49.5054557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4avvr5s/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5054791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5055019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5055251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5055482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5055772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5056017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5056226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5056458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5056564Z ok (5.638s)
2022-11-23T02:47:49.5056584Z 
2022-11-23T02:47:49.5056854Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5056967Z Ran 1 test in 5.638s
2022-11-23T02:47:49.5056986Z 
2022-11-23T02:47:49.5057080Z OK
2022-11-23T02:47:49.5057100Z 
2022-11-23T02:47:49.5057224Z Generating XML reports...
2022-11-23T02:47:49.5057687Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023921.xml
2022-11-23T02:47:49.5058062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5058223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5058599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5058788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5058811Z 
2022-11-23T02:47:49.5058920Z Running tests...
2022-11-23T02:47:49.5059185Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5059495Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5059776Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5059994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83344
2022-11-23T02:47:49.5060199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83345
2022-11-23T02:47:49.5060570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5060745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5061121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5061368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5061735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5061909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5062288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5062478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5062696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5063233Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-11-23T02:47:49.5063507Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-11-23T02:47:49.5063769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph76foc2q
2022-11-23T02:47:49.5064037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph76foc2q/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5064266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5064849Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-11-23T02:47:49.5065133Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-11-23T02:47:49.5065389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuinxo3df
2022-11-23T02:47:49.5065643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuinxo3df/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5065751Z ok (3.932s)
2022-11-23T02:47:49.5065771Z 
2022-11-23T02:47:49.5066034Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5066145Z Ran 1 test in 3.932s
2022-11-23T02:47:49.5066165Z 
2022-11-23T02:47:49.5066258Z OK
2022-11-23T02:47:49.5066277Z 
2022-11-23T02:47:49.5066401Z Generating XML reports...
2022-11-23T02:47:49.5066864Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023929.xml
2022-11-23T02:47:49.5067239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5067399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5067781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5067972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5067995Z 
2022-11-23T02:47:49.5068103Z Running tests...
2022-11-23T02:47:49.5068362Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5068674Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5068953Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5069171Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83451
2022-11-23T02:47:49.5069389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83452
2022-11-23T02:47:49.5069744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5069919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5070299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5070547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5070912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5071085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5071465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5071652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5071869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5072094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5072356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2njur5ua
2022-11-23T02:47:49.5072625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2njur5ua/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5072882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt5q2c2p
2022-11-23T02:47:49.5073150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt5q2c2p/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5073252Z ok (5.352s)
2022-11-23T02:47:49.5073272Z 
2022-11-23T02:47:49.5073538Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5073650Z Ran 1 test in 5.352s
2022-11-23T02:47:49.5073670Z 
2022-11-23T02:47:49.5073745Z OK
2022-11-23T02:47:49.5073764Z 
2022-11-23T02:47:49.5073939Z Generating XML reports...
2022-11-23T02:47:49.5074413Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023936.xml
2022-11-23T02:47:49.5074783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5074958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5075340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5075530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5075550Z 
2022-11-23T02:47:49.5075657Z Running tests...
2022-11-23T02:47:49.5075902Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5076212Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5076502Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5076719Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83572
2022-11-23T02:47:49.5076936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83573
2022-11-23T02:47:49.5077303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5077482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5077860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5078049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5078395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5078571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5078945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5079132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5079363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5079648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5079904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4j5vybt8
2022-11-23T02:47:49.5080172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4j5vybt8/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5080409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqdgrdhz
2022-11-23T02:47:49.5080679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqdgrdhz/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5080785Z ok (5.339s)
2022-11-23T02:47:49.5080806Z 
2022-11-23T02:47:49.5081075Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5081193Z Ran 1 test in 5.340s
2022-11-23T02:47:49.5081213Z 
2022-11-23T02:47:49.5081308Z OK
2022-11-23T02:47:49.5081327Z 
2022-11-23T02:47:49.5081452Z Generating XML reports...
2022-11-23T02:47:49.5081919Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023943.xml
2022-11-23T02:47:49.5082291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5082449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5082825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5083014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5083095Z 
2022-11-23T02:47:49.5083212Z Running tests...
2022-11-23T02:47:49.5083480Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5083792Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5084059Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5084284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83693
2022-11-23T02:47:49.5084484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83694
2022-11-23T02:47:49.5084854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5085030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5085406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5085598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5085965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5086139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5086509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5086700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5086913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5087140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5087395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dd1ywm9
2022-11-23T02:47:49.5087667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dd1ywm9/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5087917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2j_v4_xx
2022-11-23T02:47:49.5088180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2j_v4_xx/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5088413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5088697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5088913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5089148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5089248Z ok (5.944s)
2022-11-23T02:47:49.5089268Z 
2022-11-23T02:47:49.5089536Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5089647Z Ran 1 test in 5.944s
2022-11-23T02:47:49.5089666Z 
2022-11-23T02:47:49.5089761Z OK
2022-11-23T02:47:49.5089780Z 
2022-11-23T02:47:49.5089903Z Generating XML reports...
2022-11-23T02:47:49.5090367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023951.xml
2022-11-23T02:47:49.5090739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5090903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5091281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5091701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5091723Z 
2022-11-23T02:47:49.5091841Z Running tests...
2022-11-23T02:47:49.5092113Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5092504Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5092821Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5093588Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82632 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.629s)
2022-11-23T02:47:49.5093612Z 
2022-11-23T02:47:49.5093873Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5093968Z Ran 1 test in 1.629s
2022-11-23T02:47:49.5094003Z 
2022-11-23T02:47:49.5094093Z OK (skipped=1)
2022-11-23T02:47:49.5094112Z 
2022-11-23T02:47:49.5094235Z Generating XML reports...
2022-11-23T02:47:49.5094693Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023959.xml
2022-11-23T02:47:49.5095068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5095242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5095621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5095815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5095835Z 
2022-11-23T02:47:49.5095943Z Running tests...
2022-11-23T02:47:49.5096187Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5096498Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5096795Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5097541Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/83301 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.647s)
2022-11-23T02:47:49.5097562Z 
2022-11-23T02:47:49.5097826Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5098014Z Ran 1 test in 1.647s
2022-11-23T02:47:49.5098034Z 
2022-11-23T02:47:49.5098143Z OK (skipped=1)
2022-11-23T02:47:49.5098162Z 
2022-11-23T02:47:49.5098288Z Generating XML reports...
2022-11-23T02:47:49.5098751Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024003.xml
2022-11-23T02:47:49.5099119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5099276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5099656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5099847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5099867Z 
2022-11-23T02:47:49.5099974Z Running tests...
2022-11-23T02:47:49.5100237Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5100552Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5100849Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5101588Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82385 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.638s)
2022-11-23T02:47:49.5101608Z 
2022-11-23T02:47:49.5101919Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5102037Z Ran 1 test in 1.638s
2022-11-23T02:47:49.5102057Z 
2022-11-23T02:47:49.5102148Z OK (skipped=1)
2022-11-23T02:47:49.5102167Z 
2022-11-23T02:47:49.5102292Z Generating XML reports...
2022-11-23T02:47:49.5102755Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024007.xml
2022-11-23T02:47:49.5103133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5103310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5103684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5103874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5103893Z 
2022-11-23T02:47:49.5104004Z Running tests...
2022-11-23T02:47:49.5104250Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5104563Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5104880Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5105624Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82979 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.619s)
2022-11-23T02:47:49.5105645Z 
2022-11-23T02:47:49.5105905Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5106018Z Ran 1 test in 1.619s
2022-11-23T02:47:49.5106038Z 
2022-11-23T02:47:49.5106145Z OK (skipped=1)
2022-11-23T02:47:49.5106164Z 
2022-11-23T02:47:49.5106291Z Generating XML reports...
2022-11-23T02:47:49.5106745Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024011.xml
2022-11-23T02:47:49.5107114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5107273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5107713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5107905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5107925Z 
2022-11-23T02:47:49.5108034Z Running tests...
2022-11-23T02:47:49.5108298Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5108607Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5108921Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5109664Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82400 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.613s)
2022-11-23T02:47:49.5109689Z 
2022-11-23T02:47:49.5109952Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5110064Z Ran 1 test in 1.613s
2022-11-23T02:47:49.5110084Z 
2022-11-23T02:47:49.5110173Z OK (skipped=1)
2022-11-23T02:47:49.5110192Z 
2022-11-23T02:47:49.5110315Z Generating XML reports...
2022-11-23T02:47:49.5110772Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024015.xml
2022-11-23T02:47:49.5111187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5111372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5111756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5111947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5111971Z 
2022-11-23T02:47:49.5112079Z Running tests...
2022-11-23T02:47:49.5112324Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5112638Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5112951Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5113691Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82500 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2022-11-23T02:47:49.5113712Z 
2022-11-23T02:47:49.5113972Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5114082Z Ran 1 test in 1.637s
2022-11-23T02:47:49.5114105Z 
2022-11-23T02:47:49.5114211Z OK (skipped=1)
2022-11-23T02:47:49.5114230Z 
2022-11-23T02:47:49.5114355Z Generating XML reports...
2022-11-23T02:47:49.5114808Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024019.xml
2022-11-23T02:47:49.5115177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5115335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5115716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5115906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5115926Z 
2022-11-23T02:47:49.5116033Z Running tests...
2022-11-23T02:47:49.5116296Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5116605Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5116906Z   test_fp16 (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5117128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84029
2022-11-23T02:47:49.5117346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84030
2022-11-23T02:47:49.5117703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5117882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5118255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5118447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5118808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5118980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5119350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5119537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5119751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5119970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5120273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbsnot8o
2022-11-23T02:47:49.5120555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbsnot8o/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5120812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjocyl4t
2022-11-23T02:47:49.5121088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjocyl4t/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5121194Z ok (5.831s)
2022-11-23T02:47:49.5121215Z 
2022-11-23T02:47:49.5121480Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5121574Z Ran 1 test in 5.832s
2022-11-23T02:47:49.5121610Z 
2022-11-23T02:47:49.5121687Z OK
2022-11-23T02:47:49.5121706Z 
2022-11-23T02:47:49.5121830Z Generating XML reports...
2022-11-23T02:47:49.5122288Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024023.xml
2022-11-23T02:47:49.5122660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5122834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5123271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5123470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5123492Z 
2022-11-23T02:47:49.5123601Z Running tests...
2022-11-23T02:47:49.5123847Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5124158Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5124441Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5124658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84150
2022-11-23T02:47:49.5124879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84151
2022-11-23T02:47:49.5125249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5125424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5125801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5126048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5126398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5126573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5126951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5127143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5127373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5127923Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5128149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5128726Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5129034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2za2pasa
2022-11-23T02:47:49.5129315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2za2pasa/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5129554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmov7ro_
2022-11-23T02:47:49.5129828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmov7ro_/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5129933Z ok (5.333s)
2022-11-23T02:47:49.5129954Z 
2022-11-23T02:47:49.5130223Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5130335Z Ran 1 test in 5.334s
2022-11-23T02:47:49.5130355Z 
2022-11-23T02:47:49.5130450Z OK
2022-11-23T02:47:49.5130469Z 
2022-11-23T02:47:49.5130593Z Generating XML reports...
2022-11-23T02:47:49.5131060Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024031.xml
2022-11-23T02:47:49.5131650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5131829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5132217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5132414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5132434Z 
2022-11-23T02:47:49.5132544Z Running tests...
2022-11-23T02:47:49.5132810Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5133118Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5133399Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5133625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84271
2022-11-23T02:47:49.5133825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84272
2022-11-23T02:47:49.5134198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5134374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5134851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5135044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5135411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5135588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5135961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5136156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5136371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5136919Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5137152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5137740Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5138010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7prjfuks
2022-11-23T02:47:49.5138285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7prjfuks/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5138540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzvh3ues
2022-11-23T02:47:49.5138814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzvh3ues/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5138919Z ok (5.431s)
2022-11-23T02:47:49.5138939Z 
2022-11-23T02:47:49.5139208Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5139304Z Ran 1 test in 5.431s
2022-11-23T02:47:49.5139342Z 
2022-11-23T02:47:49.5139419Z OK
2022-11-23T02:47:49.5139437Z 
2022-11-23T02:47:49.5139561Z Generating XML reports...
2022-11-23T02:47:49.5140028Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024039.xml
2022-11-23T02:47:49.5140402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5140578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5140958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5141154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5141174Z 
2022-11-23T02:47:49.5141283Z Running tests...
2022-11-23T02:47:49.5141530Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5141841Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5142104Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5142325Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84392
2022-11-23T02:47:49.5142541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84393
2022-11-23T02:47:49.5142911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5143087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5143523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5143698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5144065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5144237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5144615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5144803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5145034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5145260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5145515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivzn348t
2022-11-23T02:47:49.5145790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivzn348t/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5146028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenc4hkjt
2022-11-23T02:47:49.5146294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenc4hkjt/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5146397Z ok (5.799s)
2022-11-23T02:47:49.5146417Z 
2022-11-23T02:47:49.5146683Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5146854Z Ran 1 test in 5.799s
2022-11-23T02:47:49.5146876Z 
2022-11-23T02:47:49.5146975Z OK
2022-11-23T02:47:49.5146995Z 
2022-11-23T02:47:49.5147121Z Generating XML reports...
2022-11-23T02:47:49.5147583Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024047.xml
2022-11-23T02:47:49.5147936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5148115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5148489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5148680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5148699Z 
2022-11-23T02:47:49.5148807Z Running tests...
2022-11-23T02:47:49.5149067Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5149383Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5149696Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5149914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84513
2022-11-23T02:47:49.5150120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84514
2022-11-23T02:47:49.5150490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5150664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5151039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5151228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5151594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5151767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5152138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5152310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5152599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5152827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5153085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwx0r6m9
2022-11-23T02:47:49.5153353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwx0r6m9/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5153605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6g6_pt5
2022-11-23T02:47:49.5153870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6g6_pt5/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5154103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5154339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5154559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5154794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5155019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5155246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5155474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5155744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5155978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5156202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5156412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5156644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5156870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5157092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5157311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5157536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5157760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5157985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5158212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5158421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5158646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5158868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5159090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5159314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5159537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5159760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5159983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5160190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5160469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5160690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5160915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5161143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5161365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5161589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5161808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5162015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5183547Z ok (6.797s)
2022-11-23T02:47:49.5183579Z 
2022-11-23T02:47:49.5183930Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5184061Z Ran 1 test in 6.798s
2022-11-23T02:47:49.5184082Z 
2022-11-23T02:47:49.5184168Z OK
2022-11-23T02:47:49.5184187Z 
2022-11-23T02:47:49.5184310Z Generating XML reports...
2022-11-23T02:47:49.5184781Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024055.xml
2022-11-23T02:47:49.5185149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5185485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5185887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5186075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5186096Z 
2022-11-23T02:47:49.5186188Z Running tests...
2022-11-23T02:47:49.5186448Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5186759Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5187042Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5187257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84634
2022-11-23T02:47:49.5187469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84635
2022-11-23T02:47:49.5187845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5188011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5188375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5188563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5188931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5189099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5189458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5189640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5189869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5190092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5190236Z skip: Need at least 4 CUDA devices (3.920s)
2022-11-23T02:47:49.5190257Z 
2022-11-23T02:47:49.5190505Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5190613Z Ran 1 test in 3.920s
2022-11-23T02:47:49.5190633Z 
2022-11-23T02:47:49.5190875Z OK (skipped=1)
2022-11-23T02:47:49.5190895Z 
2022-11-23T02:47:49.5191009Z Generating XML reports...
2022-11-23T02:47:49.5191473Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024104.xml
2022-11-23T02:47:49.5191837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5192007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5192379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5192555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5192581Z 
2022-11-23T02:47:49.5192673Z Running tests...
2022-11-23T02:47:49.5192925Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5193232Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5193506Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5193727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84737
2022-11-23T02:47:49.5193935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84738
2022-11-23T02:47:49.5194296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5194470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5194882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5195071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5195433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5195608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5195971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5196153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5196382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5196922Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5197457Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5198002Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5198529Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5199063Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5199648Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5199878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5200408Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5200940Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5201515Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5202043Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5202573Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5203102Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5203203Z ok (3.935s)
2022-11-23T02:47:49.5203224Z 
2022-11-23T02:47:49.5203481Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5203591Z Ran 1 test in 3.935s
2022-11-23T02:47:49.5203611Z 
2022-11-23T02:47:49.5203699Z OK
2022-11-23T02:47:49.5203720Z 
2022-11-23T02:47:49.5203844Z Generating XML reports...
2022-11-23T02:47:49.5204303Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024110.xml
2022-11-23T02:47:49.5204667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5204847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5205218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5205393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5205423Z 
2022-11-23T02:47:49.5205515Z Running tests...
2022-11-23T02:47:49.5205778Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5206141Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5206426Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5206646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84840
2022-11-23T02:47:49.5206855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84841
2022-11-23T02:47:49.5207222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5207398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5207762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5207946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5208307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5208479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5208849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5209030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5209254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5209539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5209786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphx1n6gb7
2022-11-23T02:47:49.5210047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphx1n6gb7/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5210296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0k3kmeq
2022-11-23T02:47:49.5210567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0k3kmeq/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5210662Z ok (5.837s)
2022-11-23T02:47:49.5210683Z 
2022-11-23T02:47:49.5210944Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5211056Z Ran 1 test in 5.837s
2022-11-23T02:47:49.5211076Z 
2022-11-23T02:47:49.5211162Z OK
2022-11-23T02:47:49.5211181Z 
2022-11-23T02:47:49.5211300Z Generating XML reports...
2022-11-23T02:47:49.5211999Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024116.xml
2022-11-23T02:47:49.5212379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5212545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5212916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5213110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5213130Z 
2022-11-23T02:47:49.5213230Z Running tests...
2022-11-23T02:47:49.5213487Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5213789Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5214084Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5214308Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84961
2022-11-23T02:47:49.5214527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84962
2022-11-23T02:47:49.5214896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5215163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5215544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5215734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5216096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5216267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5216632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5216820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5217047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5217279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5217539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp432xvqfy
2022-11-23T02:47:49.5217810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp432xvqfy/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5218064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nn2g667
2022-11-23T02:47:49.5218329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nn2g667/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5218415Z ok (5.830s)
2022-11-23T02:47:49.5218451Z 
2022-11-23T02:47:49.5218776Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5218900Z Ran 1 test in 5.831s
2022-11-23T02:47:49.5218919Z 
2022-11-23T02:47:49.5219014Z OK
2022-11-23T02:47:49.5219033Z 
2022-11-23T02:47:49.5219156Z Generating XML reports...
2022-11-23T02:47:49.5219622Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024125.xml
2022-11-23T02:47:49.5219995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5220171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5220549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5220725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5220747Z 
2022-11-23T02:47:49.5220855Z Running tests...
2022-11-23T02:47:49.5221121Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5221433Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5221741Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5221965Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85082
2022-11-23T02:47:49.5222185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85083
2022-11-23T02:47:49.5222556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5222714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5223094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5223287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5223651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5223824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5224196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5224443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5224678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5224906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5225150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmj3ygk15
2022-11-23T02:47:49.5225419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmj3ygk15/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5225677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplakmzm_k
2022-11-23T02:47:49.5225947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplakmzm_k/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5226182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5226419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5226521Z ok (5.799s)
2022-11-23T02:47:49.5226541Z 
2022-11-23T02:47:49.5226812Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5226907Z Ran 1 test in 5.799s
2022-11-23T02:47:49.5226944Z 
2022-11-23T02:47:49.5227019Z OK
2022-11-23T02:47:49.5227038Z 
2022-11-23T02:47:49.5227161Z Generating XML reports...
2022-11-23T02:47:49.5227625Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024133.xml
2022-11-23T02:47:49.5228060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5228244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5228625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5228868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5228889Z 
2022-11-23T02:47:49.5228999Z Running tests...
2022-11-23T02:47:49.5229244Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5229554Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5229866Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5230092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85203
2022-11-23T02:47:49.5230313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85204
2022-11-23T02:47:49.5230684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5230857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5231238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5231410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5231773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5231946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5232322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5232512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5232744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5232974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5233229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9suhdqv
2022-11-23T02:47:49.5233559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9suhdqv/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5233797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cp9ka5g
2022-11-23T02:47:49.5234065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cp9ka5g/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5234300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5234542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5234645Z ok (5.940s)
2022-11-23T02:47:49.5234664Z 
2022-11-23T02:47:49.5234935Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5235048Z Ran 1 test in 5.941s
2022-11-23T02:47:49.5235068Z 
2022-11-23T02:47:49.5235163Z OK
2022-11-23T02:47:49.5235182Z 
2022-11-23T02:47:49.5235287Z Generating XML reports...
2022-11-23T02:47:49.5235752Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024141.xml
2022-11-23T02:47:49.5236123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5236297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5236674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5236910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5236932Z 
2022-11-23T02:47:49.5237048Z Running tests...
2022-11-23T02:47:49.5237311Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5237622Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5237882Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5238106Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85324
2022-11-23T02:47:49.5238323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85325
2022-11-23T02:47:49.5238694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5238867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5239249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5239438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5239806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5239963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5240343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5240533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5240762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5240989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5241138Z skip: Need at least 4 CUDA devices (3.927s)
2022-11-23T02:47:49.5241158Z 
2022-11-23T02:47:49.5241425Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5241537Z Ran 1 test in 3.927s
2022-11-23T02:47:49.5241556Z 
2022-11-23T02:47:49.5241663Z OK (skipped=1)
2022-11-23T02:47:49.5241682Z 
2022-11-23T02:47:49.5241788Z Generating XML reports...
2022-11-23T02:47:49.5242246Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024149.xml
2022-11-23T02:47:49.5242678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5242852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5243228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5243418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5243438Z 
2022-11-23T02:47:49.5243546Z Running tests...
2022-11-23T02:47:49.5243808Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5244102Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5244381Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5244598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85427
2022-11-23T02:47:49.5244820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85428
2022-11-23T02:47:49.5245189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5245364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5245742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5245977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5246350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5246507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5246884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5247075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5247304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5247523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5247672Z skip: Need at least 8 CUDA devices (3.927s)
2022-11-23T02:47:49.5247691Z 
2022-11-23T02:47:49.5247954Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5248068Z Ran 1 test in 3.927s
2022-11-23T02:47:49.5248088Z 
2022-11-23T02:47:49.5248181Z OK (skipped=1)
2022-11-23T02:47:49.5248218Z 
2022-11-23T02:47:49.5248325Z Generating XML reports...
2022-11-23T02:47:49.5248785Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024155.xml
2022-11-23T02:47:49.5249154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5249331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5249709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5249898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5249917Z 
2022-11-23T02:47:49.5250025Z Running tests...
2022-11-23T02:47:49.5250286Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5250583Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5250883Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5251103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85530
2022-11-23T02:47:49.5251323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85531
2022-11-23T02:47:49.5252019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5252197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5252574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5252764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5253133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5253288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5253666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5253854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5254087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5254307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5254412Z ok (4.726s)
2022-11-23T02:47:49.5254434Z 
2022-11-23T02:47:49.5254698Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5254810Z Ran 1 test in 4.726s
2022-11-23T02:47:49.5254830Z 
2022-11-23T02:47:49.5254905Z OK
2022-11-23T02:47:49.5254924Z 
2022-11-23T02:47:49.5255049Z Generating XML reports...
2022-11-23T02:47:49.5255588Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024202.xml
2022-11-23T02:47:49.5255973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5256147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5256523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5256714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5256734Z 
2022-11-23T02:47:49.5256844Z Running tests...
2022-11-23T02:47:49.5257089Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5257401Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5257712Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5257931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85639
2022-11-23T02:47:49.5258146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85640
2022-11-23T02:47:49.5258513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5258689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5259067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5259257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5259606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5259778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5260152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5260340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5260570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5260798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5261026Z skip: Need at least 4 CUDA devices (3.906s)
2022-11-23T02:47:49.5261046Z 
2022-11-23T02:47:49.5261311Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5261406Z Ran 1 test in 3.906s
2022-11-23T02:47:49.5261443Z 
2022-11-23T02:47:49.5261532Z OK (skipped=1)
2022-11-23T02:47:49.5261551Z 
2022-11-23T02:47:49.5261675Z Generating XML reports...
2022-11-23T02:47:49.5262136Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024209.xml
2022-11-23T02:47:49.5262505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5262681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5263058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5263256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5263275Z 
2022-11-23T02:47:49.5263383Z Running tests...
2022-11-23T02:47:49.5263631Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5263943Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5264254Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5264521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85742
2022-11-23T02:47:49.5264745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85743
2022-11-23T02:47:49.5265115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5265290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5265672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5265863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5266212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5266386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5266759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5266952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5267182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5267409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5267668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpmru681
2022-11-23T02:47:49.5267942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpmru681/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5268182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk37jf29z
2022-11-23T02:47:49.5268449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk37jf29z/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5268684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5268921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5269023Z ok (5.824s)
2022-11-23T02:47:49.5269043Z 
2022-11-23T02:47:49.5269310Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5269422Z Ran 1 test in 5.824s
2022-11-23T02:47:49.5269442Z 
2022-11-23T02:47:49.5269534Z OK
2022-11-23T02:47:49.5269554Z 
2022-11-23T02:47:49.5269675Z Generating XML reports...
2022-11-23T02:47:49.5270185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024215.xml
2022-11-23T02:47:49.5270558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5270733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5271111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5271307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5271326Z 
2022-11-23T02:47:49.5271434Z Running tests...
2022-11-23T02:47:49.5271698Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5272008Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5272305Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5272529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85863
2022-11-23T02:47:49.5272745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85864
2022-11-23T02:47:49.5273116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5273290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5273703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5273887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5274267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5274456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5274818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5275005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5275233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5275455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5275713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2c1ermx
2022-11-23T02:47:49.5275988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2c1ermx/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5276239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4v0kwblh
2022-11-23T02:47:49.5276505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4v0kwblh/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5276727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5276958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5277060Z ok (5.831s)
2022-11-23T02:47:49.5277080Z 
2022-11-23T02:47:49.5277345Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5277457Z Ran 1 test in 5.832s
2022-11-23T02:47:49.5277476Z 
2022-11-23T02:47:49.5277570Z OK
2022-11-23T02:47:49.5277589Z 
2022-11-23T02:47:49.5277714Z Generating XML reports...
2022-11-23T02:47:49.5278178Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024223.xml
2022-11-23T02:47:49.5278547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5278705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5279080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5279330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5279349Z 
2022-11-23T02:47:49.5279458Z Running tests...
2022-11-23T02:47:49.5279720Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5280030Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5280315Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5280534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85984
2022-11-23T02:47:49.5280735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85985
2022-11-23T02:47:49.5281108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5281286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5281663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5281851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5282216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5282389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5282807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5283002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5283216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5283445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5283552Z ok (21.268s)
2022-11-23T02:47:49.5283572Z 
2022-11-23T02:47:49.5283840Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5283952Z Ran 1 test in 21.268s
2022-11-23T02:47:49.5283972Z 
2022-11-23T02:47:49.5284065Z OK
2022-11-23T02:47:49.5284084Z 
2022-11-23T02:47:49.5284208Z Generating XML reports...
2022-11-23T02:47:49.5284668Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024231.xml
2022-11-23T02:47:49.5285025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5285201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5285579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5285769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5285792Z 
2022-11-23T02:47:49.5285901Z Running tests...
2022-11-23T02:47:49.5286164Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5286474Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5286657Z   test_no_grad (__main__.DistributedDataParallelTest)
2022-11-23T02:47:49.5286894Z Note: this test can be sped up by only running it on a CPU module ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5287117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86105
2022-11-23T02:47:49.5287334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86106
2022-11-23T02:47:49.5287702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5287879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5288316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5288505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5288865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5289038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5289396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5289588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5289819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5290048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5290309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpkt8dwq
2022-11-23T02:47:49.5290586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpkt8dwq/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5290841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmgjnf58
2022-11-23T02:47:49.5291109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmgjnf58/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5291194Z ok (5.747s)
2022-11-23T02:47:49.5291230Z 
2022-11-23T02:47:49.5291700Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5291912Z Ran 1 test in 5.748s
2022-11-23T02:47:49.5291936Z 
2022-11-23T02:47:49.5292037Z OK
2022-11-23T02:47:49.5292057Z 
2022-11-23T02:47:49.5292184Z Generating XML reports...
2022-11-23T02:47:49.5292657Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024255.xml
2022-11-23T02:47:49.5293028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5293206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5293583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5293757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5293776Z 
2022-11-23T02:47:49.5293883Z Running tests...
2022-11-23T02:47:49.5294147Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5294462Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5294745Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5294964Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86222
2022-11-23T02:47:49.5295182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86223
2022-11-23T02:47:49.5295555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5295715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5296095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5296284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5296656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5296828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5297198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5297385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5297691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5297919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5298158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2p911v3s
2022-11-23T02:47:49.5298426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2p911v3s/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5298678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwk54csi3
2022-11-23T02:47:49.5298948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwk54csi3/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5299050Z ok (5.329s)
2022-11-23T02:47:49.5299070Z 
2022-11-23T02:47:49.5299338Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5299451Z Ran 1 test in 5.329s
2022-11-23T02:47:49.5299471Z 
2022-11-23T02:47:49.5299568Z OK
2022-11-23T02:47:49.5299587Z 
2022-11-23T02:47:49.5299693Z Generating XML reports...
2022-11-23T02:47:49.5300151Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024303.xml
2022-11-23T02:47:49.5300520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5300696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5301116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5301313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5301334Z 
2022-11-23T02:47:49.5301445Z Running tests...
2022-11-23T02:47:49.5301706Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5302020Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5302274Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5302495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86339
2022-11-23T02:47:49.5302711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86340
2022-11-23T02:47:49.5303079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5303253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5303634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5303826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5304187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5304346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5304721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5304911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5305143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5305389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5305614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5305851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5306251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5306645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5306784Z ok (3.931s)
2022-11-23T02:47:49.5306803Z 
2022-11-23T02:47:49.5307074Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5307189Z Ran 1 test in 3.931s
2022-11-23T02:47:49.5307209Z 
2022-11-23T02:47:49.5307301Z OK
2022-11-23T02:47:49.5307320Z 
2022-11-23T02:47:49.5307443Z Generating XML reports...
2022-11-23T02:47:49.5307900Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024311.xml
2022-11-23T02:47:49.5308272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5308448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5308825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5309002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5309022Z 
2022-11-23T02:47:49.5309130Z Running tests...
2022-11-23T02:47:49.5309392Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5309700Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5309979Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5310198Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86446
2022-11-23T02:47:49.5310461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86447
2022-11-23T02:47:49.5310845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5311004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5311383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5311576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5311940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5312113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5312483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5312674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5312905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5313451Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5313678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5314198Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5314457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zgqdjzr
2022-11-23T02:47:49.5314727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zgqdjzr/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5314981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iae8sj4
2022-11-23T02:47:49.5315305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iae8sj4/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5315846Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5316388Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5316921Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5317462Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5318033Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5318564Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5319099Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5319637Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5320176Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5320700Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5321233Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5321808Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5322350Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5322878Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5322985Z ok (5.459s)
2022-11-23T02:47:49.5323005Z 
2022-11-23T02:47:49.5323278Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5323388Z Ran 1 test in 5.460s
2022-11-23T02:47:49.5323408Z 
2022-11-23T02:47:49.5323501Z OK
2022-11-23T02:47:49.5323521Z 
2022-11-23T02:47:49.5323643Z Generating XML reports...
2022-11-23T02:47:49.5324184Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024317.xml
2022-11-23T02:47:49.5324573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5324751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5325129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5325325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5325345Z 
2022-11-23T02:47:49.5325454Z Running tests...
2022-11-23T02:47:49.5325717Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5326027Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5326310Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5326531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86567
2022-11-23T02:47:49.5326751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86568
2022-11-23T02:47:49.5327119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5327296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5327675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5327869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5328232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5328405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5328817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5329006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5329241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5329783Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5330067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5330611Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5330869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfom587a_
2022-11-23T02:47:49.5331138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfom587a_/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5331397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpave5ww
2022-11-23T02:47:49.5331893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpave5ww/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5332440Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5333042Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5333593Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5334124Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5334665Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5335198Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5335726Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5336247Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5336846Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5337373Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5337903Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5338441Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:47:49.5339019Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5339569Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:47:49.5339673Z ok (5.431s)
2022-11-23T02:47:49.5339693Z 
2022-11-23T02:47:49.5339957Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5340069Z Ran 1 test in 5.431s
2022-11-23T02:47:49.5340088Z 
2022-11-23T02:47:49.5340181Z OK
2022-11-23T02:47:49.5340204Z 
2022-11-23T02:47:49.5340329Z Generating XML reports...
2022-11-23T02:47:49.5340792Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024325.xml
2022-11-23T02:47:49.5341162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5341343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5341722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5341914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5341934Z 
2022-11-23T02:47:49.5342025Z Running tests...
2022-11-23T02:47:49.5342287Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5342596Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5342879Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5343102Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86688
2022-11-23T02:47:49.5343318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86689
2022-11-23T02:47:49.5343749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5343923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5344285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5344479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5344843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5345020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5345390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5345581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5345812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5346043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5346301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tqaskgp
2022-11-23T02:47:49.5346558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tqaskgp/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5346810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiywyl_sq
2022-11-23T02:47:49.5347141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiywyl_sq/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5347384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5347620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5347723Z ok (6.227s)
2022-11-23T02:47:49.5347743Z 
2022-11-23T02:47:49.5348012Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5348128Z Ran 1 test in 6.227s
2022-11-23T02:47:49.5348148Z 
2022-11-23T02:47:49.5348224Z OK
2022-11-23T02:47:49.5348243Z 
2022-11-23T02:47:49.5348367Z Generating XML reports...
2022-11-23T02:47:49.5348837Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024333.xml
2022-11-23T02:47:49.5349207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5349387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5349767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5349961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5349980Z 
2022-11-23T02:47:49.5350090Z Running tests...
2022-11-23T02:47:49.5350354Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5350655Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5350935Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5351155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86809
2022-11-23T02:47:49.5351372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86810
2022-11-23T02:47:49.5351745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5351919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5352302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5352491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5352901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5353076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5353447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5353637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5353867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5354097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5354352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpem6f0k_6
2022-11-23T02:47:49.5354621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpem6f0k_6/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5354874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5df2_4o
2022-11-23T02:47:49.5355126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5df2_4o/_remote_module_non_scriptable.py
2022-11-23T02:47:49.5355360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5355591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:47:49.5355694Z ok (5.892s)
2022-11-23T02:47:49.5355714Z 
2022-11-23T02:47:49.5355978Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5356143Z Ran 1 test in 5.892s
2022-11-23T02:47:49.5356164Z 
2022-11-23T02:47:49.5356260Z OK
2022-11-23T02:47:49.5356279Z 
2022-11-23T02:47:49.5356402Z Generating XML reports...
2022-11-23T02:47:49.5356850Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024341.xml
2022-11-23T02:47:49.5357221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5357400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5357779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5357972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5357991Z 
2022-11-23T02:47:49.5358100Z Running tests...
2022-11-23T02:47:49.5358364Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5358678Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5358944Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5359149Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86930
2022-11-23T02:47:49.5359367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86931
2022-11-23T02:47:49.5359585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86932
2022-11-23T02:47:49.5359957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5360131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5360509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5360704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5361065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5361223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5361596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5361843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5362204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5362379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5362754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5362940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5363175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5363386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:47:49.5363614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5363765Z skip: Need at least 3 CUDA devices (3.923s)
2022-11-23T02:47:49.5363787Z 
2022-11-23T02:47:49.5364053Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5364168Z Ran 1 test in 3.923s
2022-11-23T02:47:49.5364188Z 
2022-11-23T02:47:49.5364295Z OK (skipped=1)
2022-11-23T02:47:49.5364314Z 
2022-11-23T02:47:49.5364437Z Generating XML reports...
2022-11-23T02:47:49.5364867Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024349.xml
2022-11-23T02:47:49.5365281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5365445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5365825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5366017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5366040Z 
2022-11-23T02:47:49.5366149Z Running tests...
2022-11-23T02:47:49.5366408Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5366717Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5366984Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5367204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87067
2022-11-23T02:47:49.5367405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87068
2022-11-23T02:47:49.5367625Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87069
2022-11-23T02:47:49.5367999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5368171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5368547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5368741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5369105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5369277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5369646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5369823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5370182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5370352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5370716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5370961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5371192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5371412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5371913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:47:49.5372049Z skip: Need at least 3 CUDA devices (4.018s)
2022-11-23T02:47:49.5372088Z 
2022-11-23T02:47:49.5372347Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5372460Z Ran 1 test in 4.018s
2022-11-23T02:47:49.5372480Z 
2022-11-23T02:47:49.5372587Z OK (skipped=1)
2022-11-23T02:47:49.5372606Z 
2022-11-23T02:47:49.5372732Z Generating XML reports...
2022-11-23T02:47:49.5373166Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024356.xml
2022-11-23T02:47:49.5373537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5373711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5374085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5374258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5374294Z 
2022-11-23T02:47:49.5374386Z Running tests...
2022-11-23T02:47:49.5374728Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5375052Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5375394Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: Frequently times out see https://github.com/pytorch/pytorch/issues/58920 (0.000s)
2022-11-23T02:47:49.5375418Z 
2022-11-23T02:47:49.5375674Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5375787Z Ran 1 test in 0.001s
2022-11-23T02:47:49.5375807Z 
2022-11-23T02:47:49.5375912Z OK (skipped=1)
2022-11-23T02:47:49.5375931Z 
2022-11-23T02:47:49.5376055Z Generating XML reports...
2022-11-23T02:47:49.5376467Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024402.xml
2022-11-23T02:47:49.5376836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5377015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5377389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5377579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5377598Z 
2022-11-23T02:47:49.5377713Z Running tests...
2022-11-23T02:47:49.5377974Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5378285Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5378554Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5378757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87237
2022-11-23T02:47:49.5378976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87238
2022-11-23T02:47:49.5379195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87239
2022-11-23T02:47:49.5379563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5379739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5380112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5380373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5380737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5380896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5381276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5381469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5381826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5381999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5382371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5382562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5382792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5383004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5383231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:47:49.5383380Z skip: Need at least 3 CUDA devices (4.037s)
2022-11-23T02:47:49.5383400Z 
2022-11-23T02:47:49.5383707Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5383826Z Ran 1 test in 4.038s
2022-11-23T02:47:49.5383845Z 
2022-11-23T02:47:49.5383953Z OK (skipped=1)
2022-11-23T02:47:49.5383972Z 
2022-11-23T02:47:49.5384096Z Generating XML reports...
2022-11-23T02:47:49.5384529Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024404.xml
2022-11-23T02:47:49.5384901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5385059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5385434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5385625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5385645Z 
2022-11-23T02:47:49.5385754Z Running tests...
2022-11-23T02:47:49.5386021Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5386332Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5386603Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5386824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87374
2022-11-23T02:47:49.5387045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87375
2022-11-23T02:47:49.5387244Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87376
2022-11-23T02:47:49.5387612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5387787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5388165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5388354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5388717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5388892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5389318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5389488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5389847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5390020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5390396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5390585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5390815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5391044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5391270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:47:49.5391357Z ok (4.030s)
2022-11-23T02:47:49.5391393Z 
2022-11-23T02:47:49.5391645Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5391757Z Ran 1 test in 4.030s
2022-11-23T02:47:49.5391776Z 
2022-11-23T02:47:49.5391870Z OK
2022-11-23T02:47:49.5391890Z 
2022-11-23T02:47:49.5392012Z Generating XML reports...
2022-11-23T02:47:49.5392440Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024411.xml
2022-11-23T02:47:49.5392847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5393028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5393405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5393575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5393598Z 
2022-11-23T02:47:49.5393707Z Running tests...
2022-11-23T02:47:49.5393970Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5394280Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5394543Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5394760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87511
2022-11-23T02:47:49.5394980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87512
2022-11-23T02:47:49.5395196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87513
2022-11-23T02:47:49.5395550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5395726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5396110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5396301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5396666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5396839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5397215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5397403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5397761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5397916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5398283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5398528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5398759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:47:49.5398979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5399204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5399307Z ok (3.925s)
2022-11-23T02:47:49.5399327Z 
2022-11-23T02:47:49.5399598Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5399693Z Ran 1 test in 3.925s
2022-11-23T02:47:49.5399729Z 
2022-11-23T02:47:49.5399805Z OK
2022-11-23T02:47:49.5399824Z 
2022-11-23T02:47:49.5399950Z Generating XML reports...
2022-11-23T02:47:49.5400383Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024417.xml
2022-11-23T02:47:49.5400749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5400923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5401294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5401484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5401503Z 
2022-11-23T02:47:49.5401611Z Running tests...
2022-11-23T02:47:49.5401905Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5402231Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5402497Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5402716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87648
2022-11-23T02:47:49.5402937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87649
2022-11-23T02:47:49.5403151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87650
2022-11-23T02:47:49.5403514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5403690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5404054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5404243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5404611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5404787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5405162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5405349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5405712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5405886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5406253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5406427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5406659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5406888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:47:49.5407114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5407278Z ok (3.895s)
2022-11-23T02:47:49.5407298Z 
2022-11-23T02:47:49.5407565Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5407676Z Ran 1 test in 3.895s
2022-11-23T02:47:49.5407695Z 
2022-11-23T02:47:49.5407789Z OK
2022-11-23T02:47:49.5407809Z 
2022-11-23T02:47:49.5407916Z Generating XML reports...
2022-11-23T02:47:49.5408342Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024423.xml
2022-11-23T02:47:49.5408711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5408886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5409258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5409448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5409471Z 
2022-11-23T02:47:49.5409580Z Running tests...
2022-11-23T02:47:49.5409841Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5410136Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5410403Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: Test does not pass when run locally (0.001s)
2022-11-23T02:47:49.5410422Z 
2022-11-23T02:47:49.5410682Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5410855Z Ran 1 test in 0.001s
2022-11-23T02:47:49.5410876Z 
2022-11-23T02:47:49.5410989Z OK (skipped=1)
2022-11-23T02:47:49.5411008Z 
2022-11-23T02:47:49.5411131Z Generating XML reports...
2022-11-23T02:47:49.5411787Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024430.xml
2022-11-23T02:47:49.5412167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5412347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5412703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5412894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5412914Z 
2022-11-23T02:47:49.5413023Z Running tests...
2022-11-23T02:47:49.5413283Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5413597Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5413837Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5414056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87818
2022-11-23T02:47:49.5414272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87819
2022-11-23T02:47:49.5414492Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87820
2022-11-23T02:47:49.5414848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5415025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5415401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5415593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5415958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5416130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5416499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5416777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5417122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5417293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5417660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5417850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5418084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5418306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:47:49.5418531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5418680Z skip: Need at least 3 CUDA devices (3.941s)
2022-11-23T02:47:49.5418704Z 
2022-11-23T02:47:49.5418970Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5419065Z Ran 1 test in 3.941s
2022-11-23T02:47:49.5419084Z 
2022-11-23T02:47:49.5419191Z OK (skipped=1)
2022-11-23T02:47:49.5419210Z 
2022-11-23T02:47:49.5419332Z Generating XML reports...
2022-11-23T02:47:49.5419761Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024432.xml
2022-11-23T02:47:49.5420194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5420379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5420760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5420950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5420974Z 
2022-11-23T02:47:49.5421082Z Running tests...
2022-11-23T02:47:49.5421330Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5421638Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5421969Z   test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5422189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87955
2022-11-23T02:47:49.5422558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5422731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5423161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5423356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5423571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5423819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5424220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5424322Z ok (4.736s)
2022-11-23T02:47:49.5424342Z 
2022-11-23T02:47:49.5424603Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5424715Z Ran 1 test in 4.736s
2022-11-23T02:47:49.5424735Z 
2022-11-23T02:47:49.5424830Z OK
2022-11-23T02:47:49.5424850Z 
2022-11-23T02:47:49.5424973Z Generating XML reports...
2022-11-23T02:47:49.5425508Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024438.xml
2022-11-23T02:47:49.5425878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5426115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5426494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5426685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5426704Z 
2022-11-23T02:47:49.5426813Z Running tests...
2022-11-23T02:47:49.5427073Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5427387Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5427722Z   test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5427925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88030
2022-11-23T02:47:49.5428295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5428472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5428901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5429092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5429323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5429619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5430028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5430772Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:47:49.5430873Z   warnings.warn(
2022-11-23T02:47:49.5430974Z ok (4.705s)
2022-11-23T02:47:49.5430995Z 
2022-11-23T02:47:49.5431260Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5431374Z Ran 1 test in 4.705s
2022-11-23T02:47:49.5431394Z 
2022-11-23T02:47:49.5431488Z OK
2022-11-23T02:47:49.5431507Z 
2022-11-23T02:47:49.5431635Z Generating XML reports...
2022-11-23T02:47:49.5432185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024445.xml
2022-11-23T02:47:49.5432555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5432730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5433091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5433287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5433306Z 
2022-11-23T02:47:49.5433414Z Running tests...
2022-11-23T02:47:49.5433675Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5433986Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5434308Z   test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5434530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88105
2022-11-23T02:47:49.5434903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5435060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5435438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5435686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5435915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5436161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5436562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5436665Z ok (4.590s)
2022-11-23T02:47:49.5436688Z 
2022-11-23T02:47:49.5436951Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5437063Z Ran 1 test in 4.590s
2022-11-23T02:47:49.5437082Z 
2022-11-23T02:47:49.5437158Z OK
2022-11-23T02:47:49.5437176Z 
2022-11-23T02:47:49.5437300Z Generating XML reports...
2022-11-23T02:47:49.5437847Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024452.xml
2022-11-23T02:47:49.5438220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5438397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5438772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5438962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5438982Z 
2022-11-23T02:47:49.5439142Z Running tests...
2022-11-23T02:47:49.5439395Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5439708Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5440040Z   test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5440264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88180
2022-11-23T02:47:49.5440632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5440807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5441187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5441379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5441613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5441842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5442246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5442351Z ok (4.597s)
2022-11-23T02:47:49.5442371Z 
2022-11-23T02:47:49.5442635Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5442747Z Ran 1 test in 4.598s
2022-11-23T02:47:49.5442767Z 
2022-11-23T02:47:49.5442859Z OK
2022-11-23T02:47:49.5442879Z 
2022-11-23T02:47:49.5443001Z Generating XML reports...
2022-11-23T02:47:49.5443549Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024459.xml
2022-11-23T02:47:49.5443920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5444080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5444459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5444650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5444725Z 
2022-11-23T02:47:49.5444840Z Running tests...
2022-11-23T02:47:49.5445105Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5445415Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5445677Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: GPUs are available, skipping test (0.001s)
2022-11-23T02:47:49.5445697Z 
2022-11-23T02:47:49.5445956Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5446051Z Ran 1 test in 0.001s
2022-11-23T02:47:49.5446089Z 
2022-11-23T02:47:49.5446180Z OK (skipped=1)
2022-11-23T02:47:49.5446200Z 
2022-11-23T02:47:49.5446322Z Generating XML reports...
2022-11-23T02:47:49.5446772Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123024506.xml
2022-11-23T02:47:49.5447139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5447319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5447695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5447885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5447905Z 
2022-11-23T02:47:49.5448013Z Running tests...
2022-11-23T02:47:49.5448258Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5448618Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5448883Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5449102Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88288
2022-11-23T02:47:49.5449320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88289
2022-11-23T02:47:49.5449696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5449871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5450250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5450423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5450791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5450966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5451338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5451770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5452014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5452261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5452485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5452729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5453122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5453525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5453628Z ok (4.751s)
2022-11-23T02:47:49.5453648Z 
2022-11-23T02:47:49.5453912Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5454024Z Ran 1 test in 4.751s
2022-11-23T02:47:49.5454045Z 
2022-11-23T02:47:49.5454228Z OK
2022-11-23T02:47:49.5454248Z 
2022-11-23T02:47:49.5454373Z Generating XML reports...
2022-11-23T02:47:49.5454807Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024508.xml
2022-11-23T02:47:49.5455175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5455334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5455719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5455909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5455929Z 
2022-11-23T02:47:49.5456037Z Running tests...
2022-11-23T02:47:49.5456302Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5456612Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5456868Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5457088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88397
2022-11-23T02:47:49.5457289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88398
2022-11-23T02:47:49.5457658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5457833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5458274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5458473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5458836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5459012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5459389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5459575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5459786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5460032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5460257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5460496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5460896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5461291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5461397Z ok (5.324s)
2022-11-23T02:47:49.5461417Z 
2022-11-23T02:47:49.5461679Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5461773Z Ran 1 test in 5.324s
2022-11-23T02:47:49.5461808Z 
2022-11-23T02:47:49.5461884Z OK
2022-11-23T02:47:49.5461903Z 
2022-11-23T02:47:49.5462027Z Generating XML reports...
2022-11-23T02:47:49.5462458Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024515.xml
2022-11-23T02:47:49.5462826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5463000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5463380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5463659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5463678Z 
2022-11-23T02:47:49.5463787Z Running tests...
2022-11-23T02:47:49.5464034Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5464345Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5464587Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5464810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88514
2022-11-23T02:47:49.5465030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88515
2022-11-23T02:47:49.5465400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5465575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5465955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5466132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5466496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5466670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5467040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5467266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5467502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5467749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5467971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5468218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5468605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5468999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5469101Z ok (5.341s)
2022-11-23T02:47:49.5469120Z 
2022-11-23T02:47:49.5469384Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5469498Z Ran 1 test in 5.341s
2022-11-23T02:47:49.5469518Z 
2022-11-23T02:47:49.5469612Z OK
2022-11-23T02:47:49.5469631Z 
2022-11-23T02:47:49.5469754Z Generating XML reports...
2022-11-23T02:47:49.5470185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024523.xml
2022-11-23T02:47:49.5470535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5470714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5471092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5471282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5471302Z 
2022-11-23T02:47:49.5471409Z Running tests...
2022-11-23T02:47:49.5471674Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5471984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5472226Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5472444Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88631
2022-11-23T02:47:49.5472644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88632
2022-11-23T02:47:49.5473072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5473247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5473623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5473813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5474181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5474354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5474730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5474901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5475133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5475378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5475604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5475839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5476292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5476695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5476797Z ok (5.340s)
2022-11-23T02:47:49.5476817Z 
2022-11-23T02:47:49.5477081Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5477176Z Ran 1 test in 5.340s
2022-11-23T02:47:49.5477199Z 
2022-11-23T02:47:49.5477295Z OK
2022-11-23T02:47:49.5477315Z 
2022-11-23T02:47:49.5477440Z Generating XML reports...
2022-11-23T02:47:49.5477871Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024531.xml
2022-11-23T02:47:49.5478239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5478417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5478798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5478988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5479008Z 
2022-11-23T02:47:49.5479115Z Running tests...
2022-11-23T02:47:49.5479362Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5479670Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5479906Z   test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5480124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88748
2022-11-23T02:47:49.5480342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88749
2022-11-23T02:47:49.5480712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5480889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5481270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5481444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5481808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5482032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5482407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5482597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5482827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5483073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5483303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5483548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5483933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5484333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5484436Z ok (5.391s)
2022-11-23T02:47:49.5484456Z 
2022-11-23T02:47:49.5484719Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5484832Z Ran 1 test in 5.391s
2022-11-23T02:47:49.5484852Z 
2022-11-23T02:47:49.5484944Z OK
2022-11-23T02:47:49.5484964Z 
2022-11-23T02:47:49.5485087Z Generating XML reports...
2022-11-23T02:47:49.5485566Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024538.xml
2022-11-23T02:47:49.5485929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5486105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5486482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5486678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5486697Z 
2022-11-23T02:47:49.5486805Z Running tests...
2022-11-23T02:47:49.5487063Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5487376Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5487622Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5487842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88865
2022-11-23T02:47:49.5488045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88866
2022-11-23T02:47:49.5488416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5488590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5488968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5489161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5489525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5489695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5490068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5490243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5490475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5490717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5490941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5491236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5491969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5492385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5492488Z ok (5.345s)
2022-11-23T02:47:49.5492508Z 
2022-11-23T02:47:49.5492775Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5492870Z Ran 1 test in 5.345s
2022-11-23T02:47:49.5492890Z 
2022-11-23T02:47:49.5492982Z OK
2022-11-23T02:47:49.5493001Z 
2022-11-23T02:47:49.5493126Z Generating XML reports...
2022-11-23T02:47:49.5493556Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024546.xml
2022-11-23T02:47:49.5493929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5494106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5494480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5494671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5494690Z 
2022-11-23T02:47:49.5494798Z Running tests...
2022-11-23T02:47:49.5495042Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5495432Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5495686Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5495906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88982
2022-11-23T02:47:49.5496124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88983
2022-11-23T02:47:49.5496499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5496674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5497051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5497225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5497589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5497764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5498134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5498320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5498552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5498800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5499021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5499248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5499651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5500047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5500150Z ok (5.334s)
2022-11-23T02:47:49.5500170Z 
2022-11-23T02:47:49.5500433Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5500619Z Ran 1 test in 5.334s
2022-11-23T02:47:49.5500639Z 
2022-11-23T02:47:49.5500733Z OK
2022-11-23T02:47:49.5500752Z 
2022-11-23T02:47:49.5500876Z Generating XML reports...
2022-11-23T02:47:49.5501313Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024554.xml
2022-11-23T02:47:49.5501665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5501842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5502227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5502418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5502437Z 
2022-11-23T02:47:49.5502545Z Running tests...
2022-11-23T02:47:49.5502808Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5503117Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5503364Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5503567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89099
2022-11-23T02:47:49.5503784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89100
2022-11-23T02:47:49.5504156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5504382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5504769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5504959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5505324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5505501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5505872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5506043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5506272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5506515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5506747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5506985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5507384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5507783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5507887Z ok (4.816s)
2022-11-23T02:47:49.5507906Z 
2022-11-23T02:47:49.5508168Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5508263Z Ran 1 test in 4.816s
2022-11-23T02:47:49.5508282Z 
2022-11-23T02:47:49.5508374Z OK
2022-11-23T02:47:49.5508394Z 
2022-11-23T02:47:49.5508516Z Generating XML reports...
2022-11-23T02:47:49.5508949Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024601.xml
2022-11-23T02:47:49.5509314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5509490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5509863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5510111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5510131Z 
2022-11-23T02:47:49.5510224Z Running tests...
2022-11-23T02:47:49.5510490Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5510802Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5511040Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5511262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89208
2022-11-23T02:47:49.5511481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89209
2022-11-23T02:47:49.5511849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5512023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5512401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5512575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5512943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5513117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5513489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5513723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5513960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5514207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5514431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5514658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5515061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5515463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5515570Z ok (5.300s)
2022-11-23T02:47:49.5515590Z 
2022-11-23T02:47:49.5515856Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5515969Z Ran 1 test in 5.300s
2022-11-23T02:47:49.5515993Z 
2022-11-23T02:47:49.5516088Z OK
2022-11-23T02:47:49.5516108Z 
2022-11-23T02:47:49.5516232Z Generating XML reports...
2022-11-23T02:47:49.5516660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024609.xml
2022-11-23T02:47:49.5517017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5517190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5517565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5517755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5517775Z 
2022-11-23T02:47:49.5517882Z Running tests...
2022-11-23T02:47:49.5518147Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5518459Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5518701Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5518902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89329
2022-11-23T02:47:49.5519172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89330
2022-11-23T02:47:49.5519544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5519719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5520097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5520288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5520653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5520827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5521200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5521372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5521607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5521835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5522082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5522327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5522774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5523181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5523284Z ok (9.792s)
2022-11-23T02:47:49.5523303Z 
2022-11-23T02:47:49.5523568Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5523668Z Ran 1 test in 9.793s
2022-11-23T02:47:49.5523687Z 
2022-11-23T02:47:49.5523780Z OK
2022-11-23T02:47:49.5523799Z 
2022-11-23T02:47:49.5523925Z Generating XML reports...
2022-11-23T02:47:49.5524354Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024616.xml
2022-11-23T02:47:49.5524719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5524895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5525275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5525466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5525486Z 
2022-11-23T02:47:49.5525579Z Running tests...
2022-11-23T02:47:49.5525843Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5526157Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5526415Z   test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5526634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89450
2022-11-23T02:47:49.5526851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89451
2022-11-23T02:47:49.5527219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5527396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5527758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5527949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5528312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5528541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5528967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5529155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5529384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5529634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5529854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5530082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5530484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5530885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5530990Z ok (4.704s)
2022-11-23T02:47:49.5531010Z 
2022-11-23T02:47:49.5531277Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5531390Z Ran 1 test in 4.705s
2022-11-23T02:47:49.5531409Z 
2022-11-23T02:47:49.5531723Z OK
2022-11-23T02:47:49.5531744Z 
2022-11-23T02:47:49.5531873Z Generating XML reports...
2022-11-23T02:47:49.5532391Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024628.xml
2022-11-23T02:47:49.5532755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5532932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5533307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5533503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5533522Z 
2022-11-23T02:47:49.5533631Z Running tests...
2022-11-23T02:47:49.5533893Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5534201Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5534439Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5534646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89562
2022-11-23T02:47:49.5534863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89563
2022-11-23T02:47:49.5535233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5535407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5535791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5535982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5536352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5536525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5536900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5537072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5537301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5537545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5537849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5538093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5538499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5538895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5538999Z ok (5.327s)
2022-11-23T02:47:49.5539019Z 
2022-11-23T02:47:49.5539276Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5539389Z Ran 1 test in 5.327s
2022-11-23T02:47:49.5539408Z 
2022-11-23T02:47:49.5539502Z OK
2022-11-23T02:47:49.5539522Z 
2022-11-23T02:47:49.5539644Z Generating XML reports...
2022-11-23T02:47:49.5540075Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024635.xml
2022-11-23T02:47:49.5540449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5540623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5541000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5541192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5541212Z 
2022-11-23T02:47:49.5541303Z Running tests...
2022-11-23T02:47:49.5541627Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5541950Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5542217Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5542436Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89679
2022-11-23T02:47:49.5542657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89680
2022-11-23T02:47:49.5543026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5543203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5543564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5543755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5544124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5544296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5544672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5544867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5545097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5545343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5545565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5545788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5546193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5546587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5546690Z ok (4.707s)
2022-11-23T02:47:49.5546710Z 
2022-11-23T02:47:49.5546974Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5547139Z Ran 1 test in 4.708s
2022-11-23T02:47:49.5547160Z 
2022-11-23T02:47:49.5547252Z OK
2022-11-23T02:47:49.5547271Z 
2022-11-23T02:47:49.5547395Z Generating XML reports...
2022-11-23T02:47:49.5547810Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024643.xml
2022-11-23T02:47:49.5548179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5548354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5548735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5548925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5548945Z 
2022-11-23T02:47:49.5549053Z Running tests...
2022-11-23T02:47:49.5549315Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5549630Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5549889Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5550092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89788
2022-11-23T02:47:49.5550310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89789
2022-11-23T02:47:49.5550729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5550914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5551293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5551483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5551849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5552018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5552376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5552565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5552795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5553043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5553266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5553501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5553900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5554301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5554406Z ok (5.333s)
2022-11-23T02:47:49.5554427Z 
2022-11-23T02:47:49.5554679Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5554790Z Ran 1 test in 5.333s
2022-11-23T02:47:49.5554809Z 
2022-11-23T02:47:49.5554902Z OK
2022-11-23T02:47:49.5554922Z 
2022-11-23T02:47:49.5555045Z Generating XML reports...
2022-11-23T02:47:49.5555479Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024650.xml
2022-11-23T02:47:49.5555846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5556021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5556458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5556648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5556667Z 
2022-11-23T02:47:49.5556758Z Running tests...
2022-11-23T02:47:49.5557020Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5557328Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5557582Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5557801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89905
2022-11-23T02:47:49.5558018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89906
2022-11-23T02:47:49.5558382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5558560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5558921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5559111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5559470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5559643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5560067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5560261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5560491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5560736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5560961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5561189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5561591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5561985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5562092Z ok (5.446s)
2022-11-23T02:47:49.5562115Z 
2022-11-23T02:47:49.5562377Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5562488Z Ran 1 test in 5.446s
2022-11-23T02:47:49.5562508Z 
2022-11-23T02:47:49.5562601Z OK
2022-11-23T02:47:49.5562620Z 
2022-11-23T02:47:49.5562743Z Generating XML reports...
2022-11-23T02:47:49.5563158Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024658.xml
2022-11-23T02:47:49.5563535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5563711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5564086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5564277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5564297Z 
2022-11-23T02:47:49.5564409Z Running tests...
2022-11-23T02:47:49.5564673Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5564985Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5565230Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5565493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90022
2022-11-23T02:47:49.5565713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90023
2022-11-23T02:47:49.5566081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5566255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5566633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5566826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5567186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5567356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5567711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5567902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5568132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5568383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5568610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5568848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5569302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5569704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5569808Z ok (4.722s)
2022-11-23T02:47:49.5569829Z 
2022-11-23T02:47:49.5570082Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5570196Z Ran 1 test in 4.722s
2022-11-23T02:47:49.5570216Z 
2022-11-23T02:47:49.5570310Z OK
2022-11-23T02:47:49.5570331Z 
2022-11-23T02:47:49.5570454Z Generating XML reports...
2022-11-23T02:47:49.5570883Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024705.xml
2022-11-23T02:47:49.5571249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5571648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5572047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5572240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5572261Z 
2022-11-23T02:47:49.5572354Z Running tests...
2022-11-23T02:47:49.5572620Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5572930Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5573166Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5573383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90131
2022-11-23T02:47:49.5573600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90132
2022-11-23T02:47:49.5573968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5574144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5574502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5574690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5575146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5575319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5575691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5575879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5576107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5576355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5576581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5576802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5577206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5577605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5577710Z ok (5.322s)
2022-11-23T02:47:49.5577730Z 
2022-11-23T02:47:49.5577993Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5578105Z Ran 1 test in 5.322s
2022-11-23T02:47:49.5578126Z 
2022-11-23T02:47:49.5578219Z OK
2022-11-23T02:47:49.5578238Z 
2022-11-23T02:47:49.5578427Z Generating XML reports...
2022-11-23T02:47:49.5578856Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024713.xml
2022-11-23T02:47:49.5579227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5579403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5579784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5579974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5579993Z 
2022-11-23T02:47:49.5580102Z Running tests...
2022-11-23T02:47:49.5580364Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5580674Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5580922Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5581124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90252
2022-11-23T02:47:49.5581345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90253
2022-11-23T02:47:49.5581714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5581890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5582265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5582456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5582821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5582994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5583351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5583538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5583766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5584011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5584296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5584539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5584944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5585338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5585444Z ok (10.060s)
2022-11-23T02:47:49.5585465Z 
2022-11-23T02:47:49.5585712Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5585827Z Ran 1 test in 10.060s
2022-11-23T02:47:49.5585846Z 
2022-11-23T02:47:49.5585938Z OK
2022-11-23T02:47:49.5585957Z 
2022-11-23T02:47:49.5586082Z Generating XML reports...
2022-11-23T02:47:49.5586516Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024720.xml
2022-11-23T02:47:49.5586882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5587057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5587437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5587613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5587695Z 
2022-11-23T02:47:49.5587794Z Running tests...
2022-11-23T02:47:49.5588057Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5588368Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5588603Z   test_send_recv (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5588824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90373
2022-11-23T02:47:49.5589040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90374
2022-11-23T02:47:49.5589408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5589583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5589943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5590135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5590497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5590671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5591042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5591235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5591466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:47:49.5591714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5591919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:47:49.5592162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:47:49.5592561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5592955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:47:49.5593059Z ok (4.783s)
2022-11-23T02:47:49.5593133Z 
2022-11-23T02:47:49.5593402Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5593515Z Ran 1 test in 4.783s
2022-11-23T02:47:49.5593535Z 
2022-11-23T02:47:49.5593628Z OK
2022-11-23T02:47:49.5593647Z 
2022-11-23T02:47:49.5593772Z Generating XML reports...
2022-11-23T02:47:49.5594188Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024733.xml
2022-11-23T02:47:49.5594556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5594733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5595115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5595305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5595325Z 
2022-11-23T02:47:49.5595435Z Running tests...
2022-11-23T02:47:49.5595700Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5596011Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5596241Z   test_common_errors (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5596470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5596865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5597155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5597555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5597793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5598187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5598429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5598816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5598917Z ok (1.602s)
2022-11-23T02:47:49.5598937Z 
2022-11-23T02:47:49.5599183Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5599299Z Ran 1 test in 1.603s
2022-11-23T02:47:49.5599318Z 
2022-11-23T02:47:49.5599411Z OK
2022-11-23T02:47:49.5599431Z 
2022-11-23T02:47:49.5599558Z Generating XML reports...
2022-11-23T02:47:49.5599976Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123024740.xml
2022-11-23T02:47:49.5600343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:47:49.5600522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:47:49.5600897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:47:49.5601071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:47:49.5601107Z 
2022-11-23T02:47:49.5601201Z Running tests...
2022-11-23T02:47:49.5601464Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5601776Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:47:49.5602013Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:47:49.5602257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5602650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5602950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:47:49.5603349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:47:49.5603435Z ok (4.638s)
2022-11-23T02:47:49.5603454Z 
2022-11-23T02:47:49.5603717Z ----------------------------------------------------------------------
2022-11-23T02:47:49.5603829Z Ran 1 test in 4.638s
2022-11-23T02:47:49.5603852Z 
2022-11-23T02:47:49.5603945Z OK
2022-11-23T02:47:49.5603964Z 
2022-11-23T02:47:49.5604088Z Generating XML reports...
2022-11-23T02:47:49.5604485Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123024744.xml
2022-11-23T02:47:49.5604505Z 
2022-11-23T02:47:49.5604935Z ##[endgroup]
2022-11-23T02:47:49.5605349Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_b3knhzl_)
2022-11-23T02:47:49.5605391Z 
2022-11-23T02:47:49.7440580Z 
2022-11-23T02:47:49.7441371Z real	88m54.971s
2022-11-23T02:47:49.7441926Z user	140m38.068s
2022-11-23T02:47:49.7442380Z sys	80m18.607s
2022-11-23T02:47:49.7442638Z + assert_git_not_dirty
2022-11-23T02:47:49.7443148Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *rocm* ]]
2022-11-23T02:47:49.7443588Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *xla* ]]
2022-11-23T02:47:49.7444820Z ++ git status --porcelain
2022-11-23T02:47:50.7282449Z + git_status=
2022-11-23T02:47:50.7283276Z + [[ -n '' ]]
2022-11-23T02:47:50.7283726Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *cuda* ]]
2022-11-23T02:47:50.7284030Z + [[ 2 == 1 ]]
2022-11-23T02:47:50.7284239Z + [[ 2 == 1 ]]
2022-11-23T02:47:50.7353066Z Prepare all required actions
2022-11-23T02:47:50.7353511Z Getting action download info
2022-11-23T02:47:50.8903744Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2022-11-23T02:47:51.2333787Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-11-23T02:47:51.2334099Z with:
2022-11-23T02:47:51.2334566Z   github-token: ***
2022-11-23T02:47:51.2334812Z env:
2022-11-23T02:47:51.2335035Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:51.2335302Z   GPU_FLAG: --gpus all
2022-11-23T02:47:51.2335670Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:51.2336006Z ##[endgroup]
2022-11-23T02:47:51.2368200Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2022-11-23T02:47:51.2368529Z with:
2022-11-23T02:47:51.2368758Z   shell: bash
2022-11-23T02:47:51.2368987Z   timeout_minutes: 10
2022-11-23T02:47:51.2369237Z   max_attempts: 5
2022-11-23T02:47:51.2369528Z   retry_wait_seconds: 30
2022-11-23T02:47:51.2370026Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2022-11-23T02:47:51.2370543Z   polling_interval_seconds: 1
2022-11-23T02:47:51.2370820Z   warning_on_retry: true
2022-11-23T02:47:51.2371088Z   continue_on_error: false
2022-11-23T02:47:51.2371318Z env:
2022-11-23T02:47:51.2372263Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:51.2372549Z   GPU_FLAG: --gpus all
2022-11-23T02:47:51.2372899Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:51.2373404Z   GITHUB_TOKEN: ***
2022-11-23T02:47:51.2373657Z ##[endgroup]
2022-11-23T02:47:51.3043228Z + python3 -m pip install requests==2.26.0
2022-11-23T02:47:51.6064646Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:47:51.6317936Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2022-11-23T02:47:51.6506883Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2022-11-23T02:47:51.6534534Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.12)
2022-11-23T02:47:51.6757991Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24)
2022-11-23T02:47:51.6770395Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2022-11-23T02:47:51.9272309Z ++ python3 .github/scripts/get_workflow_job_id.py 3528293554 i-0b134addf6413be7c
2022-11-23T02:47:55.1536330Z + GHA_WORKFLOW_JOB_ID=9655199165
2022-11-23T02:47:55.1538728Z + echo job-id=9655199165
2022-11-23T02:47:55.3049924Z Command completed after 1 attempt(s).
2022-11-23T02:47:55.3183119Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-11-23T02:47:55.3183485Z kill "$MONITOR_SCRIPT_PID"
2022-11-23T02:47:55.3196703Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:47:55.3197004Z env:
2022-11-23T02:47:55.3197252Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:55.3197504Z   GPU_FLAG: --gpus all
2022-11-23T02:47:55.3197870Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:55.3198240Z   MONITOR_SCRIPT_PID: 12995
2022-11-23T02:47:55.3198483Z ##[endgroup]
2022-11-23T02:47:55.3298888Z Prepare all required actions
2022-11-23T02:47:55.3299259Z Getting action download info
2022-11-23T02:47:55.4874828Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb)
2022-11-23T02:47:55.6469983Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-11-23T02:47:55.6470271Z with:
2022-11-23T02:47:55.6470630Z   file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165
2022-11-23T02:47:55.6470976Z env:
2022-11-23T02:47:55.6471212Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:55.6471477Z   GPU_FLAG: --gpus all
2022-11-23T02:47:55.6471842Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:55.6472177Z ##[endgroup]
2022-11-23T02:47:55.6502521Z ##[group]Run # Remove any previous test jsons if they exist
2022-11-23T02:47:55.6502888Z # Remove any previous test jsons if they exist
2022-11-23T02:47:55.6503204Z rm -f test-jsons-*.zip
2022-11-23T02:47:55.6503587Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-11-23T02:47:55.6515533Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:47:55.6515835Z env:
2022-11-23T02:47:55.6516079Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:55.6516334Z   GPU_FLAG: --gpus all
2022-11-23T02:47:55.6516703Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:55.6517178Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165
2022-11-23T02:47:55.6517522Z ##[endgroup]
2022-11-23T02:47:55.6635340Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2022-11-23T02:47:55.6669700Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-11-23T02:47:55.6676985Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-11-23T02:47:55.6678141Z   adding: test/.pytorch-slow-tests.json (deflated 73%)
2022-11-23T02:47:55.6688983Z   adding: test/.pytorch-disabled-tests.json (deflated 86%)
2022-11-23T02:47:55.6712587Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T02:47:55.6712984Z # Remove any previous test reports if they exist
2022-11-23T02:47:55.6713317Z rm -f test-reports-*.zip
2022-11-23T02:47:55.6713680Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2022-11-23T02:47:55.6725448Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:47:55.6725754Z env:
2022-11-23T02:47:55.6726002Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:55.6726380Z   GPU_FLAG: --gpus all
2022-11-23T02:47:55.6726751Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:55.6727233Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165
2022-11-23T02:47:55.6727596Z ##[endgroup]
2022-11-23T02:47:55.6874190Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011903.xml (deflated 43%)
2022-11-23T02:47:55.6875129Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011905.xml (deflated 44%)
2022-11-23T02:47:55.6876017Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011907.xml (deflated 42%)
2022-11-23T02:47:55.6876904Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011915.xml (deflated 42%)
2022-11-23T02:47:55.6877803Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011923.xml (deflated 42%)
2022-11-23T02:47:55.6878652Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011930.xml (deflated 42%)
2022-11-23T02:47:55.6879708Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011940.xml (deflated 43%)
2022-11-23T02:47:55.6880613Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011942.xml (deflated 44%)
2022-11-23T02:47:55.6881496Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011944.xml (deflated 42%)
2022-11-23T02:47:55.6882376Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011952.xml (deflated 42%)
2022-11-23T02:47:55.6883237Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011959.xml (deflated 42%)
2022-11-23T02:47:55.6884119Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012007.xml (deflated 42%)
2022-11-23T02:47:55.6884948Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012017.xml (deflated 41%)
2022-11-23T02:47:55.6885742Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012026.xml (deflated 42%)
2022-11-23T02:47:55.6886514Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012028.xml (deflated 42%)
2022-11-23T02:47:55.6887303Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012035.xml (deflated 42%)
2022-11-23T02:47:55.6888086Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012039.xml (deflated 42%)
2022-11-23T02:47:55.6888870Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012041.xml (deflated 42%)
2022-11-23T02:47:55.6889665Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012044.xml (deflated 40%)
2022-11-23T02:47:55.6890430Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012053.xml (deflated 40%)
2022-11-23T02:47:55.6891205Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012101.xml (deflated 41%)
2022-11-23T02:47:55.6892329Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012109.xml (deflated 40%)
2022-11-23T02:47:55.6893250Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012118.xml (deflated 40%)
2022-11-23T02:47:55.6894014Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012127.xml (deflated 40%)
2022-11-23T02:47:55.6894791Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012135.xml (deflated 40%)
2022-11-23T02:47:55.6895557Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012143.xml (deflated 42%)
2022-11-23T02:47:55.6896330Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012147.xml (deflated 42%)
2022-11-23T02:47:55.6897089Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012154.xml (deflated 42%)
2022-11-23T02:47:55.6897864Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012159.xml (deflated 42%)
2022-11-23T02:47:55.6898639Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012207.xml (deflated 42%)
2022-11-23T02:47:55.6899497Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012209.xml (deflated 46%)
2022-11-23T02:47:55.6900277Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012218.xml (deflated 47%)
2022-11-23T02:47:55.6901071Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012226.xml (deflated 48%)
2022-11-23T02:47:55.6901836Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012235.xml (deflated 46%)
2022-11-23T02:47:55.6902613Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012244.xml (deflated 41%)
2022-11-23T02:47:55.6903366Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012246.xml (deflated 44%)
2022-11-23T02:47:55.6904148Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012249.xml (deflated 43%)
2022-11-23T02:47:55.6904915Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012251.xml (deflated 43%)
2022-11-23T02:47:55.6905685Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012253.xml (deflated 44%)
2022-11-23T02:47:55.6906436Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012256.xml (deflated 43%)
2022-11-23T02:47:55.6907214Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012258.xml (deflated 41%)
2022-11-23T02:47:55.6907986Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012301.xml (deflated 41%)
2022-11-23T02:47:55.6908752Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012310.xml (deflated 40%)
2022-11-23T02:47:55.6909505Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012319.xml (deflated 41%)
2022-11-23T02:47:55.6910277Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012321.xml (deflated 41%)
2022-11-23T02:47:55.6911042Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012323.xml (deflated 41%)
2022-11-23T02:47:55.6911806Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012332.xml (deflated 40%)
2022-11-23T02:47:55.6912643Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012341.xml (deflated 42%)
2022-11-23T02:47:55.6913410Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012346.xml (deflated 42%)
2022-11-23T02:47:55.6914181Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012350.xml (deflated 40%)
2022-11-23T02:47:55.6914945Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012358.xml (deflated 40%)
2022-11-23T02:47:55.6915708Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012407.xml (deflated 40%)
2022-11-23T02:47:55.6916462Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012415.xml (deflated 41%)
2022-11-23T02:47:55.6917234Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012418.xml (deflated 41%)
2022-11-23T02:47:55.6918033Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012420.xml (deflated 41%)
2022-11-23T02:47:55.6918857Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012422.xml (deflated 41%)
2022-11-23T02:47:55.6919631Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012425.xml (deflated 41%)
2022-11-23T02:47:55.6920401Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012427.xml (deflated 41%)
2022-11-23T02:47:55.6921168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012429.xml (deflated 41%)
2022-11-23T02:47:55.6921938Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012432.xml (deflated 41%)
2022-11-23T02:47:55.6922683Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012434.xml (deflated 41%)
2022-11-23T02:47:55.6923454Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012437.xml (deflated 42%)
2022-11-23T02:47:55.6924221Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012439.xml (deflated 41%)
2022-11-23T02:47:55.6924983Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012441.xml (deflated 42%)
2022-11-23T02:47:55.6925733Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012444.xml (deflated 41%)
2022-11-23T02:47:55.6926506Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012446.xml (deflated 41%)
2022-11-23T02:47:55.6927271Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012449.xml (deflated 41%)
2022-11-23T02:47:55.6928038Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012451.xml (deflated 41%)
2022-11-23T02:47:55.6928859Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012453.xml (deflated 41%)
2022-11-23T02:47:55.6929609Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012456.xml (deflated 41%)
2022-11-23T02:47:55.6930383Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012458.xml (deflated 41%)
2022-11-23T02:47:55.6931225Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012500.xml (deflated 42%)
2022-11-23T02:47:55.6932221Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012503.xml (deflated 41%)
2022-11-23T02:47:55.6932981Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012505.xml (deflated 42%)
2022-11-23T02:47:55.6933751Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012508.xml (deflated 41%)
2022-11-23T02:47:55.6934518Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012510.xml (deflated 42%)
2022-11-23T02:47:55.6935282Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012512.xml (deflated 43%)
2022-11-23T02:47:55.6936047Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012515.xml (deflated 43%)
2022-11-23T02:47:55.6936814Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012517.xml (deflated 41%)
2022-11-23T02:47:55.6937582Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012520.xml (deflated 41%)
2022-11-23T02:47:55.6938434Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012528.xml (deflated 41%)
2022-11-23T02:47:55.6939215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012530.xml (deflated 42%)
2022-11-23T02:47:55.6939991Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012533.xml (deflated 41%)
2022-11-23T02:47:55.6940755Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012535.xml (deflated 41%)
2022-11-23T02:47:55.6941527Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012543.xml (deflated 40%)
2022-11-23T02:47:55.6942279Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012551.xml (deflated 41%)
2022-11-23T02:47:55.6943048Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012600.xml (deflated 42%)
2022-11-23T02:47:55.6943812Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012602.xml (deflated 42%)
2022-11-23T02:47:55.6944581Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012604.xml (deflated 41%)
2022-11-23T02:47:55.6945389Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012612.xml (deflated 41%)
2022-11-23T02:47:55.6946152Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012621.xml (deflated 43%)
2022-11-23T02:47:55.6946917Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012623.xml (deflated 41%)
2022-11-23T02:47:55.6947688Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012632.xml (deflated 43%)
2022-11-23T02:47:55.6948453Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012634.xml (deflated 42%)
2022-11-23T02:47:55.6949206Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012641.xml (deflated 42%)
2022-11-23T02:47:55.6949978Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012643.xml (deflated 42%)
2022-11-23T02:47:55.6950845Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012646.xml (deflated 42%)
2022-11-23T02:47:55.6951607Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012650.xml (deflated 41%)
2022-11-23T02:47:55.6952368Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012659.xml (deflated 42%)
2022-11-23T02:47:55.6953135Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012701.xml (deflated 42%)
2022-11-23T02:47:55.6953899Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012705.xml (deflated 42%)
2022-11-23T02:47:55.6954662Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012708.xml (deflated 42%)
2022-11-23T02:47:55.6955420Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012715.xml (deflated 42%)
2022-11-23T02:47:55.6956187Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012717.xml (deflated 42%)
2022-11-23T02:47:55.6957007Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012719.xml (deflated 41%)
2022-11-23T02:47:55.6957789Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012728.xml (deflated 40%)
2022-11-23T02:47:55.6958547Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012736.xml (deflated 42%)
2022-11-23T02:47:55.6959308Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012738.xml (deflated 40%)
2022-11-23T02:47:55.6960083Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012746.xml (deflated 42%)
2022-11-23T02:47:55.6960849Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012749.xml (deflated 42%)
2022-11-23T02:47:55.6961603Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012756.xml (deflated 41%)
2022-11-23T02:47:55.6962372Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012804.xml (deflated 42%)
2022-11-23T02:47:55.6963139Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012811.xml (deflated 42%)
2022-11-23T02:47:55.6963902Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012813.xml (deflated 42%)
2022-11-23T02:47:55.6964656Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012815.xml (deflated 40%)
2022-11-23T02:47:55.6965429Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012824.xml (deflated 42%)
2022-11-23T02:47:55.6966193Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012827.xml (deflated 42%)
2022-11-23T02:47:55.6966960Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012834.xml (deflated 42%)
2022-11-23T02:47:55.6967714Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012836.xml (deflated 42%)
2022-11-23T02:47:55.6968484Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012843.xml (deflated 42%)
2022-11-23T02:47:55.6969248Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012845.xml (deflated 42%)
2022-11-23T02:47:55.6970097Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012848.xml (deflated 43%)
2022-11-23T02:47:55.6970846Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012850.xml (deflated 41%)
2022-11-23T02:47:55.6971837Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012852.xml (deflated 41%)
2022-11-23T02:47:55.6972627Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012855.xml (deflated 41%)
2022-11-23T02:47:55.6973391Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012902.xml (deflated 41%)
2022-11-23T02:47:55.6974140Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012909.xml (deflated 42%)
2022-11-23T02:47:55.6974917Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012916.xml (deflated 41%)
2022-11-23T02:47:55.6975689Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012924.xml (deflated 41%)
2022-11-23T02:47:55.6976570Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012931.xml (deflated 41%)
2022-11-23T02:47:55.6977345Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012939.xml (deflated 41%)
2022-11-23T02:47:55.6978115Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012946.xml (deflated 41%)
2022-11-23T02:47:55.6978881Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012953.xml (deflated 41%)
2022-11-23T02:47:55.6979652Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012956.xml (deflated 42%)
2022-11-23T02:47:55.6980418Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013000.xml (deflated 42%)
2022-11-23T02:47:55.6981169Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013002.xml (deflated 41%)
2022-11-23T02:47:55.6981936Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013005.xml (deflated 43%)
2022-11-23T02:47:55.6982699Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013007.xml (deflated 42%)
2022-11-23T02:47:55.6983457Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013011.xml (deflated 42%)
2022-11-23T02:47:55.6984211Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013015.xml (deflated 42%)
2022-11-23T02:47:55.6984976Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013020.xml (deflated 40%)
2022-11-23T02:47:55.6985738Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013028.xml (deflated 40%)
2022-11-23T02:47:55.6986552Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013036.xml (deflated 42%)
2022-11-23T02:47:55.6987317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013040.xml (deflated 42%)
2022-11-23T02:47:55.6988070Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml (deflated 40%)
2022-11-23T02:47:55.6988930Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013052.xml (deflated 40%)
2022-11-23T02:47:55.6989701Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013100.xml (deflated 41%)
2022-11-23T02:47:55.6990467Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013109.xml (deflated 40%)
2022-11-23T02:47:55.6991220Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013117.xml (deflated 42%)
2022-11-23T02:47:55.6991998Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013122.xml (deflated 43%)
2022-11-23T02:47:55.6992761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013124.xml (deflated 42%)
2022-11-23T02:47:55.6993530Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013128.xml (deflated 41%)
2022-11-23T02:47:55.6994281Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013137.xml (deflated 42%)
2022-11-23T02:47:55.6995044Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013141.xml (deflated 42%)
2022-11-23T02:47:55.6995867Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013145.xml (deflated 40%)
2022-11-23T02:47:55.6996659Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013154.xml (deflated 40%)
2022-11-23T02:47:55.6997409Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013203.xml (deflated 42%)
2022-11-23T02:47:55.6998175Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013208.xml (deflated 41%)
2022-11-23T02:47:55.6998948Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013216.xml (deflated 42%)
2022-11-23T02:47:55.6999711Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013219.xml (deflated 42%)
2022-11-23T02:47:55.7000465Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013221.xml (deflated 42%)
2022-11-23T02:47:55.7001235Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013223.xml (deflated 41%)
2022-11-23T02:47:55.7002004Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013226.xml (deflated 41%)
2022-11-23T02:47:55.7002768Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013228.xml (deflated 41%)
2022-11-23T02:47:55.7003522Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013231.xml (deflated 41%)
2022-11-23T02:47:55.7004292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013233.xml (deflated 41%)
2022-11-23T02:47:55.7005062Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013235.xml (deflated 41%)
2022-11-23T02:47:55.7005827Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013238.xml (deflated 41%)
2022-11-23T02:47:55.7006580Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013240.xml (deflated 42%)
2022-11-23T02:47:55.7007347Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013243.xml (deflated 42%)
2022-11-23T02:47:55.7008215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013245.xml (deflated 42%)
2022-11-23T02:47:55.7008981Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013249.xml (deflated 41%)
2022-11-23T02:47:55.7009736Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013258.xml (deflated 40%)
2022-11-23T02:47:55.7010511Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013306.xml (deflated 41%)
2022-11-23T02:47:55.7011276Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013309.xml (deflated 40%)
2022-11-23T02:47:55.7012321Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013318.xml (deflated 40%)
2022-11-23T02:47:55.7013082Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013325.xml (deflated 40%)
2022-11-23T02:47:55.7013852Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013344.xml (deflated 41%)
2022-11-23T02:47:55.7014704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013353.xml (deflated 41%)
2022-11-23T02:47:55.7015494Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013402.xml (deflated 41%)
2022-11-23T02:47:55.7016270Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013410.xml (deflated 40%)
2022-11-23T02:47:55.7017018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013419.xml (deflated 42%)
2022-11-23T02:47:55.7017789Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013423.xml (deflated 42%)
2022-11-23T02:47:55.7018557Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013427.xml (deflated 42%)
2022-11-23T02:47:55.7019324Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013436.xml (deflated 40%)
2022-11-23T02:47:55.7020078Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013444.xml (deflated 42%)
2022-11-23T02:47:55.7020849Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013448.xml (deflated 40%)
2022-11-23T02:47:55.7021612Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013457.xml (deflated 42%)
2022-11-23T02:47:55.7022381Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013501.xml (deflated 41%)
2022-11-23T02:47:55.7023130Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013510.xml (deflated 41%)
2022-11-23T02:47:55.7023893Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013518.xml (deflated 40%)
2022-11-23T02:47:55.7024661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013526.xml (deflated 42%)
2022-11-23T02:47:55.7025432Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013530.xml (deflated 42%)
2022-11-23T02:47:55.7026182Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013534.xml (deflated 42%)
2022-11-23T02:47:55.7026950Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013538.xml (deflated 40%)
2022-11-23T02:47:55.7027809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013547.xml (deflated 41%)
2022-11-23T02:47:55.7028575Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013555.xml (deflated 40%)
2022-11-23T02:47:55.7029384Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013602.xml (deflated 41%)
2022-11-23T02:47:55.7030155Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013609.xml (deflated 42%)
2022-11-23T02:47:55.7030921Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013613.xml (deflated 42%)
2022-11-23T02:47:55.7031685Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013618.xml (deflated 40%)
2022-11-23T02:47:55.7032443Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013624.xml (deflated 41%)
2022-11-23T02:47:55.7033207Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013627.xml (deflated 41%)
2022-11-23T02:47:55.7034035Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013629.xml (deflated 41%)
2022-11-23T02:47:55.7034818Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013638.xml (deflated 41%)
2022-11-23T02:47:55.7035575Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013641.xml (deflated 42%)
2022-11-23T02:47:55.7036345Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013643.xml (deflated 41%)
2022-11-23T02:47:55.7037110Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013651.xml (deflated 42%)
2022-11-23T02:47:55.7037879Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013655.xml (deflated 41%)
2022-11-23T02:47:55.7038635Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013702.xml (deflated 40%)
2022-11-23T02:47:55.7039400Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013711.xml (deflated 41%)
2022-11-23T02:47:55.7040172Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013718.xml (deflated 40%)
2022-11-23T02:47:55.7040942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013724.xml (deflated 40%)
2022-11-23T02:47:55.7041696Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013731.xml (deflated 40%)
2022-11-23T02:47:55.7042465Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013739.xml (deflated 42%)
2022-11-23T02:47:55.7043233Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013742.xml (deflated 43%)
2022-11-23T02:47:55.7044000Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013744.xml (deflated 42%)
2022-11-23T02:47:55.7044748Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013747.xml (deflated 42%)
2022-11-23T02:47:55.7045519Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013749.xml (deflated 41%)
2022-11-23T02:47:55.7046359Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013813.xml (deflated 41%)
2022-11-23T02:47:55.7047124Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013837.xml (deflated 42%)
2022-11-23T02:47:55.7047882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013840.xml (deflated 41%)
2022-11-23T02:47:55.7048736Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013842.xml (deflated 41%)
2022-11-23T02:47:55.7049487Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013844.xml (deflated 41%)
2022-11-23T02:47:55.7050252Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013847.xml (deflated 41%)
2022-11-23T02:47:55.7051018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013849.xml (deflated 41%)
2022-11-23T02:47:55.7052013Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013857.xml (deflated 41%)
2022-11-23T02:47:55.7052774Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013904.xml (deflated 41%)
2022-11-23T02:47:55.7053633Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013913.xml (deflated 41%)
2022-11-23T02:47:55.7054427Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013920.xml (deflated 42%)
2022-11-23T02:47:55.7055194Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013924.xml (deflated 42%)
2022-11-23T02:47:55.7055947Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013927.xml (deflated 42%)
2022-11-23T02:47:55.7056719Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013929.xml (deflated 41%)
2022-11-23T02:47:55.7057484Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013932.xml (deflated 40%)
2022-11-23T02:47:55.7058255Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013938.xml (deflated 40%)
2022-11-23T02:47:55.7059016Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013945.xml (deflated 41%)
2022-11-23T02:47:55.7059782Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013948.xml (deflated 42%)
2022-11-23T02:47:55.7060544Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013950.xml (deflated 42%)
2022-11-23T02:47:55.7061315Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013954.xml (deflated 40%)
2022-11-23T02:47:55.7062072Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014003.xml (deflated 41%)
2022-11-23T02:47:55.7062843Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014011.xml (deflated 40%)
2022-11-23T02:47:55.7063613Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014019.xml (deflated 42%)
2022-11-23T02:47:55.7064378Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014023.xml (deflated 41%)
2022-11-23T02:47:55.7065126Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014028.xml (deflated 40%)
2022-11-23T02:47:55.7065994Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014034.xml (deflated 40%)
2022-11-23T02:47:55.7066761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014041.xml (deflated 42%)
2022-11-23T02:47:55.7067530Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014045.xml (deflated 41%)
2022-11-23T02:47:55.7068280Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014048.xml (deflated 41%)
2022-11-23T02:47:55.7069047Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014050.xml (deflated 41%)
2022-11-23T02:47:55.7069811Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014053.xml (deflated 41%)
2022-11-23T02:47:55.7070583Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014055.xml (deflated 41%)
2022-11-23T02:47:55.7071333Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014057.xml (deflated 41%)
2022-11-23T02:47:55.7072156Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014100.xml (deflated 42%)
2022-11-23T02:47:55.7072944Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014102.xml (deflated 41%)
2022-11-23T02:47:55.7073712Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014105.xml (deflated 41%)
2022-11-23T02:47:55.7074465Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014107.xml (deflated 42%)
2022-11-23T02:47:55.7075236Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014109.xml (deflated 40%)
2022-11-23T02:47:55.7076006Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014118.xml (deflated 42%)
2022-11-23T02:47:55.7076770Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014120.xml (deflated 41%)
2022-11-23T02:47:55.7077522Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014129.xml (deflated 40%)
2022-11-23T02:47:55.7078298Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014137.xml (deflated 41%)
2022-11-23T02:47:55.7079067Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014140.xml (deflated 41%)
2022-11-23T02:47:55.7079834Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014148.xml (deflated 40%)
2022-11-23T02:47:55.7080584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014156.xml (deflated 41%)
2022-11-23T02:47:55.7081352Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014159.xml (deflated 41%)
2022-11-23T02:47:55.7082121Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014201.xml (deflated 42%)
2022-11-23T02:47:55.7082889Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014204.xml (deflated 42%)
2022-11-23T02:47:55.7083643Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014206.xml (deflated 41%)
2022-11-23T02:47:55.7084402Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014215.xml (deflated 40%)
2022-11-23T02:47:55.7085247Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014224.xml (deflated 42%)
2022-11-23T02:47:55.7086011Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014227.xml (deflated 42%)
2022-11-23T02:47:55.7086760Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014229.xml (deflated 41%)
2022-11-23T02:47:55.7087533Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014231.xml (deflated 43%)
2022-11-23T02:47:55.7088299Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014234.xml (deflated 42%)
2022-11-23T02:47:55.7089060Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014236.xml (deflated 42%)
2022-11-23T02:47:55.7089830Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014239.xml (deflated 42%)
2022-11-23T02:47:55.7090582Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014241.xml (deflated 43%)
2022-11-23T02:47:55.7091404Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014243.xml (deflated 40%)
2022-11-23T02:47:55.7092415Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014251.xml (deflated 40%)
2022-11-23T02:47:55.7093189Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014300.xml (deflated 40%)
2022-11-23T02:47:55.7093937Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014308.xml (deflated 43%)
2022-11-23T02:47:55.7094710Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014310.xml (deflated 43%)
2022-11-23T02:47:55.7095476Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014313.xml (deflated 42%)
2022-11-23T02:47:55.7096247Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014315.xml (deflated 43%)
2022-11-23T02:47:55.7096995Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014317.xml (deflated 42%)
2022-11-23T02:47:55.7097770Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014320.xml (deflated 42%)
2022-11-23T02:47:55.7098536Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014322.xml (deflated 41%)
2022-11-23T02:47:55.7099304Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014331.xml (deflated 41%)
2022-11-23T02:47:55.7100058Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014333.xml (deflated 40%)
2022-11-23T02:47:55.7100829Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014342.xml (deflated 41%)
2022-11-23T02:47:55.7101593Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014350.xml (deflated 40%)
2022-11-23T02:47:55.7102360Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014404.xml (deflated 40%)
2022-11-23T02:47:55.7103114Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014419.xml (deflated 41%)
2022-11-23T02:47:55.7103989Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014427.xml (deflated 41%)
2022-11-23T02:47:55.7104755Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014430.xml (deflated 42%)
2022-11-23T02:47:55.7105523Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014436.xml (deflated 42%)
2022-11-23T02:47:55.7106275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014441.xml (deflated 42%)
2022-11-23T02:47:55.7107036Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014443.xml (deflated 42%)
2022-11-23T02:47:55.7107805Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014445.xml (deflated 40%)
2022-11-23T02:47:55.7108573Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014454.xml (deflated 40%)
2022-11-23T02:47:55.7109317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014503.xml (deflated 40%)
2022-11-23T02:47:55.7110088Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014511.xml (deflated 39%)
2022-11-23T02:47:55.7110931Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014519.xml (deflated 39%)
2022-11-23T02:47:55.7111715Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014528.xml (deflated 40%)
2022-11-23T02:47:55.7112471Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014537.xml (deflated 40%)
2022-11-23T02:47:55.7113249Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014545.xml (deflated 42%)
2022-11-23T02:47:55.7114021Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014549.xml (deflated 41%)
2022-11-23T02:47:55.7114794Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014556.xml (deflated 42%)
2022-11-23T02:47:55.7115548Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014600.xml (deflated 42%)
2022-11-23T02:47:55.7116316Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014608.xml (deflated 41%)
2022-11-23T02:47:55.7117084Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014611.xml (deflated 46%)
2022-11-23T02:47:55.7117847Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014619.xml (deflated 47%)
2022-11-23T02:47:55.7118606Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014628.xml (deflated 48%)
2022-11-23T02:47:55.7119376Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014637.xml (deflated 45%)
2022-11-23T02:47:55.7120154Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014645.xml (deflated 41%)
2022-11-23T02:47:55.7120918Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014648.xml (deflated 43%)
2022-11-23T02:47:55.7121666Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014650.xml (deflated 43%)
2022-11-23T02:47:55.7122434Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014652.xml (deflated 43%)
2022-11-23T02:47:55.7123271Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014655.xml (deflated 43%)
2022-11-23T02:47:55.7124035Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014657.xml (deflated 43%)
2022-11-23T02:47:55.7124814Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014659.xml (deflated 41%)
2022-11-23T02:47:55.7125562Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014702.xml (deflated 40%)
2022-11-23T02:47:55.7126334Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014711.xml (deflated 40%)
2022-11-23T02:47:55.7127103Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014720.xml (deflated 41%)
2022-11-23T02:47:55.7127867Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014722.xml (deflated 41%)
2022-11-23T02:47:55.7128619Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014725.xml (deflated 40%)
2022-11-23T02:47:55.7129506Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014734.xml (deflated 40%)
2022-11-23T02:47:55.7130290Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014743.xml (deflated 42%)
2022-11-23T02:47:55.7131056Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014747.xml (deflated 42%)
2022-11-23T02:47:55.7132019Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014751.xml (deflated 40%)
2022-11-23T02:47:55.7132792Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014759.xml (deflated 40%)
2022-11-23T02:47:55.7133558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014808.xml (deflated 40%)
2022-11-23T02:47:55.7134331Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014817.xml (deflated 41%)
2022-11-23T02:47:55.7135080Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014819.xml (deflated 41%)
2022-11-23T02:47:55.7135842Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014822.xml (deflated 41%)
2022-11-23T02:47:55.7136613Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014824.xml (deflated 41%)
2022-11-23T02:47:55.7137387Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014826.xml (deflated 41%)
2022-11-23T02:47:55.7138138Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014829.xml (deflated 41%)
2022-11-23T02:47:55.7138905Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014831.xml (deflated 41%)
2022-11-23T02:47:55.7139682Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014834.xml (deflated 41%)
2022-11-23T02:47:55.7140447Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014836.xml (deflated 41%)
2022-11-23T02:47:55.7141196Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014838.xml (deflated 42%)
2022-11-23T02:47:55.7141958Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014841.xml (deflated 41%)
2022-11-23T02:47:55.7142845Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014843.xml (deflated 41%)
2022-11-23T02:47:55.7143614Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014846.xml (deflated 41%)
2022-11-23T02:47:55.7144478Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014848.xml (deflated 41%)
2022-11-23T02:47:55.7145242Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014850.xml (deflated 41%)
2022-11-23T02:47:55.7145991Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014853.xml (deflated 41%)
2022-11-23T02:47:55.7146763Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014855.xml (deflated 41%)
2022-11-23T02:47:55.7147530Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014858.xml (deflated 41%)
2022-11-23T02:47:55.7148294Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014900.xml (deflated 41%)
2022-11-23T02:47:55.7149126Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014902.xml (deflated 41%)
2022-11-23T02:47:55.7149923Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014905.xml (deflated 41%)
2022-11-23T02:47:55.7150688Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014907.xml (deflated 42%)
2022-11-23T02:47:55.7151453Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014909.xml (deflated 41%)
2022-11-23T02:47:55.7152206Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014912.xml (deflated 41%)
2022-11-23T02:47:55.7152976Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014914.xml (deflated 43%)
2022-11-23T02:47:55.7153748Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014917.xml (deflated 42%)
2022-11-23T02:47:55.7154513Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014919.xml (deflated 41%)
2022-11-23T02:47:55.7155279Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014921.xml (deflated 41%)
2022-11-23T02:47:55.7156034Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014930.xml (deflated 41%)
2022-11-23T02:47:55.7156801Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014932.xml (deflated 41%)
2022-11-23T02:47:55.7157566Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014934.xml (deflated 41%)
2022-11-23T02:47:55.7158345Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014937.xml (deflated 40%)
2022-11-23T02:47:55.7159102Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014945.xml (deflated 40%)
2022-11-23T02:47:55.7159865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014953.xml (deflated 40%)
2022-11-23T02:47:55.7160625Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015001.xml (deflated 42%)
2022-11-23T02:47:55.7161460Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015004.xml (deflated 42%)
2022-11-23T02:47:55.7162211Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015006.xml (deflated 41%)
2022-11-23T02:47:55.7162986Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015015.xml (deflated 41%)
2022-11-23T02:47:55.7163747Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015023.xml (deflated 43%)
2022-11-23T02:47:55.7164513Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015025.xml (deflated 41%)
2022-11-23T02:47:55.7165271Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015033.xml (deflated 42%)
2022-11-23T02:47:55.7166040Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015036.xml (deflated 42%)
2022-11-23T02:47:55.7166804Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015043.xml (deflated 42%)
2022-11-23T02:47:55.7167569Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015045.xml (deflated 42%)
2022-11-23T02:47:55.7168385Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015047.xml (deflated 42%)
2022-11-23T02:47:55.7169169Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015052.xml (deflated 41%)
2022-11-23T02:47:55.7169934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015101.xml (deflated 42%)
2022-11-23T02:47:55.7170700Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015103.xml (deflated 42%)
2022-11-23T02:47:55.7171632Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015107.xml (deflated 42%)
2022-11-23T02:47:55.7172422Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015110.xml (deflated 42%)
2022-11-23T02:47:55.7173240Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015116.xml (deflated 42%)
2022-11-23T02:47:55.7174010Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015119.xml (deflated 42%)
2022-11-23T02:47:55.7174762Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015121.xml (deflated 41%)
2022-11-23T02:47:55.7175525Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015129.xml (deflated 41%)
2022-11-23T02:47:55.7176297Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015137.xml (deflated 42%)
2022-11-23T02:47:55.7177059Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015140.xml (deflated 40%)
2022-11-23T02:47:55.7177814Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015148.xml (deflated 42%)
2022-11-23T02:47:55.7178584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015150.xml (deflated 42%)
2022-11-23T02:47:55.7179347Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015157.xml (deflated 41%)
2022-11-23T02:47:55.7180113Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015205.xml (deflated 41%)
2022-11-23T02:47:55.7180977Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015212.xml (deflated 42%)
2022-11-23T02:47:55.7181746Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015214.xml (deflated 42%)
2022-11-23T02:47:55.7182518Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015217.xml (deflated 41%)
2022-11-23T02:47:55.7183282Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015226.xml (deflated 42%)
2022-11-23T02:47:55.7184034Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015228.xml (deflated 42%)
2022-11-23T02:47:55.7184807Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015235.xml (deflated 42%)
2022-11-23T02:47:55.7185574Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015237.xml (deflated 42%)
2022-11-23T02:47:55.7186338Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015244.xml (deflated 42%)
2022-11-23T02:47:55.7187180Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015246.xml (deflated 42%)
2022-11-23T02:47:55.7187974Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015249.xml (deflated 42%)
2022-11-23T02:47:55.7188739Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015251.xml (deflated 41%)
2022-11-23T02:47:55.7189507Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015254.xml (deflated 41%)
2022-11-23T02:47:55.7190258Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015256.xml (deflated 41%)
2022-11-23T02:47:55.7191023Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015303.xml (deflated 41%)
2022-11-23T02:47:55.7191788Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015311.xml (deflated 41%)
2022-11-23T02:47:55.7192557Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015317.xml (deflated 41%)
2022-11-23T02:47:55.7193310Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015325.xml (deflated 40%)
2022-11-23T02:47:55.7194081Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015332.xml (deflated 40%)
2022-11-23T02:47:55.7194854Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015340.xml (deflated 41%)
2022-11-23T02:47:55.7195618Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015347.xml (deflated 41%)
2022-11-23T02:47:55.7196368Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015354.xml (deflated 41%)
2022-11-23T02:47:55.7197142Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015357.xml (deflated 42%)
2022-11-23T02:47:55.7197911Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015401.xml (deflated 41%)
2022-11-23T02:47:55.7198679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015403.xml (deflated 41%)
2022-11-23T02:47:55.7199441Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015406.xml (deflated 43%)
2022-11-23T02:47:55.7200269Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015408.xml (deflated 42%)
2022-11-23T02:47:55.7201034Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015412.xml (deflated 42%)
2022-11-23T02:47:55.7201800Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015416.xml (deflated 42%)
2022-11-23T02:47:55.7202566Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015421.xml (deflated 40%)
2022-11-23T02:47:55.7203319Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015429.xml (deflated 40%)
2022-11-23T02:47:55.7204081Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015437.xml (deflated 42%)
2022-11-23T02:47:55.7204849Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015441.xml (deflated 42%)
2022-11-23T02:47:55.7205615Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015445.xml (deflated 40%)
2022-11-23T02:47:55.7206416Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015454.xml (deflated 40%)
2022-11-23T02:47:55.7207200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015502.xml (deflated 41%)
2022-11-23T02:47:55.7207968Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015510.xml (deflated 40%)
2022-11-23T02:47:55.7208735Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015519.xml (deflated 42%)
2022-11-23T02:47:55.7209490Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015523.xml (deflated 42%)
2022-11-23T02:47:55.7210256Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015525.xml (deflated 42%)
2022-11-23T02:47:55.7211023Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015530.xml (deflated 41%)
2022-11-23T02:47:55.7211990Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015538.xml (deflated 42%)
2022-11-23T02:47:55.7212750Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015542.xml (deflated 42%)
2022-11-23T02:47:55.7213513Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015547.xml (deflated 40%)
2022-11-23T02:47:55.7214286Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015556.xml (deflated 40%)
2022-11-23T02:47:55.7215047Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015605.xml (deflated 42%)
2022-11-23T02:47:55.7215799Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015609.xml (deflated 41%)
2022-11-23T02:47:55.7216565Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015618.xml (deflated 41%)
2022-11-23T02:47:55.7217327Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015621.xml (deflated 42%)
2022-11-23T02:47:55.7218090Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015623.xml (deflated 41%)
2022-11-23T02:47:55.7218945Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015625.xml (deflated 41%)
2022-11-23T02:47:55.7219711Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015628.xml (deflated 41%)
2022-11-23T02:47:55.7220477Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015630.xml (deflated 41%)
2022-11-23T02:47:55.7221247Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015633.xml (deflated 41%)
2022-11-23T02:47:55.7222003Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015635.xml (deflated 41%)
2022-11-23T02:47:55.7222771Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015637.xml (deflated 41%)
2022-11-23T02:47:55.7223545Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015640.xml (deflated 41%)
2022-11-23T02:47:55.7224310Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015642.xml (deflated 41%)
2022-11-23T02:47:55.7225061Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015645.xml (deflated 41%)
2022-11-23T02:47:55.7225901Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015647.xml (deflated 42%)
2022-11-23T02:47:55.7226689Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015651.xml (deflated 41%)
2022-11-23T02:47:55.7227453Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015659.xml (deflated 40%)
2022-11-23T02:47:55.7228203Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015708.xml (deflated 41%)
2022-11-23T02:47:55.7229031Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015711.xml (deflated 40%)
2022-11-23T02:47:55.7229800Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015719.xml (deflated 40%)
2022-11-23T02:47:55.7230571Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015727.xml (deflated 40%)
2022-11-23T02:47:55.7231317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015746.xml (deflated 41%)
2022-11-23T02:47:55.7232086Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015754.xml (deflated 41%)
2022-11-23T02:47:55.7232855Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015803.xml (deflated 41%)
2022-11-23T02:47:55.7233622Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015812.xml (deflated 41%)
2022-11-23T02:47:55.7234367Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015820.xml (deflated 42%)
2022-11-23T02:47:55.7235140Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015825.xml (deflated 42%)
2022-11-23T02:47:55.7235903Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015829.xml (deflated 42%)
2022-11-23T02:47:55.7236666Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015837.xml (deflated 40%)
2022-11-23T02:47:55.7237417Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015846.xml (deflated 42%)
2022-11-23T02:47:55.7238266Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015850.xml (deflated 41%)
2022-11-23T02:47:55.7239032Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015858.xml (deflated 42%)
2022-11-23T02:47:55.7239806Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015902.xml (deflated 41%)
2022-11-23T02:47:55.7240572Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015911.xml (deflated 41%)
2022-11-23T02:47:55.7241321Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015919.xml (deflated 40%)
2022-11-23T02:47:55.7242084Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015927.xml (deflated 42%)
2022-11-23T02:47:55.7242861Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015932.xml (deflated 42%)
2022-11-23T02:47:55.7243624Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015936.xml (deflated 42%)
2022-11-23T02:47:55.7244426Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015940.xml (deflated 40%)
2022-11-23T02:47:55.7245212Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015949.xml (deflated 41%)
2022-11-23T02:47:55.7245974Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015957.xml (deflated 40%)
2022-11-23T02:47:55.7246738Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020004.xml (deflated 40%)
2022-11-23T02:47:55.7247503Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020011.xml (deflated 42%)
2022-11-23T02:47:55.7248270Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020015.xml (deflated 42%)
2022-11-23T02:47:55.7249035Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020019.xml (deflated 40%)
2022-11-23T02:47:55.7249808Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020026.xml (deflated 42%)
2022-11-23T02:47:55.7250559Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020028.xml (deflated 41%)
2022-11-23T02:47:55.7251320Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020031.xml (deflated 41%)
2022-11-23T02:47:55.7252586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020040.xml (deflated 42%)
2022-11-23T02:47:55.7253367Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020042.xml (deflated 42%)
2022-11-23T02:47:55.7254116Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020045.xml (deflated 40%)
2022-11-23T02:47:55.7254889Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020053.xml (deflated 42%)
2022-11-23T02:47:55.7255655Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020057.xml (deflated 42%)
2022-11-23T02:47:55.7256421Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020104.xml (deflated 40%)
2022-11-23T02:47:55.7257173Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020112.xml (deflated 41%)
2022-11-23T02:47:55.7258063Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020119.xml (deflated 40%)
2022-11-23T02:47:55.7258833Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020126.xml (deflated 40%)
2022-11-23T02:47:55.7259603Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020133.xml (deflated 40%)
2022-11-23T02:47:55.7260355Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020141.xml (deflated 43%)
2022-11-23T02:47:55.7261130Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020144.xml (deflated 43%)
2022-11-23T02:47:55.7261897Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020146.xml (deflated 42%)
2022-11-23T02:47:55.7262669Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020149.xml (deflated 42%)
2022-11-23T02:47:55.7263419Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020151.xml (deflated 41%)
2022-11-23T02:47:55.7264261Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020215.xml (deflated 41%)
2022-11-23T02:47:55.7265049Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020239.xml (deflated 42%)
2022-11-23T02:47:55.7265821Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020241.xml (deflated 42%)
2022-11-23T02:47:55.7266570Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020244.xml (deflated 42%)
2022-11-23T02:47:55.7267339Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020246.xml (deflated 41%)
2022-11-23T02:47:55.7268104Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020249.xml (deflated 41%)
2022-11-23T02:47:55.7268870Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020251.xml (deflated 41%)
2022-11-23T02:47:55.7269618Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020259.xml (deflated 41%)
2022-11-23T02:47:55.7270386Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020306.xml (deflated 41%)
2022-11-23T02:47:55.7271160Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020315.xml (deflated 41%)
2022-11-23T02:47:55.7271970Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020322.xml (deflated 42%)
2022-11-23T02:47:55.7272732Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020327.xml (deflated 42%)
2022-11-23T02:47:55.7273512Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020329.xml (deflated 42%)
2022-11-23T02:47:55.7274289Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020331.xml (deflated 41%)
2022-11-23T02:47:55.7275062Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020334.xml (deflated 40%)
2022-11-23T02:47:55.7275826Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020340.xml (deflated 40%)
2022-11-23T02:47:55.7276716Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020347.xml (deflated 41%)
2022-11-23T02:47:55.7277496Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020350.xml (deflated 42%)
2022-11-23T02:47:55.7278272Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020352.xml (deflated 42%)
2022-11-23T02:47:55.7279045Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020356.xml (deflated 40%)
2022-11-23T02:47:55.7279826Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020405.xml (deflated 41%)
2022-11-23T02:47:55.7280606Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020413.xml (deflated 40%)
2022-11-23T02:47:55.7281383Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020421.xml (deflated 42%)
2022-11-23T02:47:55.7282168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020425.xml (deflated 41%)
2022-11-23T02:47:55.7282935Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020430.xml (deflated 40%)
2022-11-23T02:47:55.7283759Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020436.xml (deflated 40%)
2022-11-23T02:47:55.7284558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020443.xml (deflated 42%)
2022-11-23T02:47:55.7285336Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020447.xml (deflated 41%)
2022-11-23T02:47:55.7286100Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020450.xml (deflated 41%)
2022-11-23T02:47:55.7286883Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020452.xml (deflated 41%)
2022-11-23T02:47:55.7287660Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020455.xml (deflated 41%)
2022-11-23T02:47:55.7288444Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020457.xml (deflated 41%)
2022-11-23T02:47:55.7289213Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020459.xml (deflated 41%)
2022-11-23T02:47:55.7290095Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020502.xml (deflated 42%)
2022-11-23T02:47:55.7290856Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020504.xml (deflated 41%)
2022-11-23T02:47:55.7291850Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020507.xml (deflated 41%)
2022-11-23T02:47:55.7292649Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020509.xml (deflated 42%)
2022-11-23T02:47:55.7293434Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020511.xml (deflated 40%)
2022-11-23T02:47:55.7294198Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020520.xml (deflated 42%)
2022-11-23T02:47:55.7294975Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020522.xml (deflated 40%)
2022-11-23T02:47:55.7295754Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020531.xml (deflated 41%)
2022-11-23T02:47:55.7296636Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020539.xml (deflated 41%)
2022-11-23T02:47:55.7297402Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020542.xml (deflated 40%)
2022-11-23T02:47:55.7298182Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020550.xml (deflated 41%)
2022-11-23T02:47:55.7298967Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020558.xml (deflated 41%)
2022-11-23T02:47:55.7299747Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020601.xml (deflated 42%)
2022-11-23T02:47:55.7300514Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020603.xml (deflated 41%)
2022-11-23T02:47:55.7301299Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020605.xml (deflated 42%)
2022-11-23T02:47:55.7302077Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020608.xml (deflated 41%)
2022-11-23T02:47:55.7302918Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020617.xml (deflated 41%)
2022-11-23T02:47:55.7303703Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020626.xml (deflated 42%)
2022-11-23T02:47:55.7304485Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020628.xml (deflated 42%)
2022-11-23T02:47:55.7305264Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020631.xml (deflated 41%)
2022-11-23T02:47:55.7306050Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020633.xml (deflated 43%)
2022-11-23T02:47:55.7306814Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020635.xml (deflated 42%)
2022-11-23T02:47:55.7307596Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020638.xml (deflated 42%)
2022-11-23T02:47:55.7308378Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020640.xml (deflated 42%)
2022-11-23T02:47:55.7309157Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020643.xml (deflated 43%)
2022-11-23T02:47:55.7309919Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020645.xml (deflated 41%)
2022-11-23T02:47:55.7310700Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020653.xml (deflated 40%)
2022-11-23T02:47:55.7311484Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020701.xml (deflated 40%)
2022-11-23T02:47:55.7312260Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020709.xml (deflated 43%)
2022-11-23T02:47:55.7313028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020712.xml (deflated 43%)
2022-11-23T02:47:55.7313810Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020714.xml (deflated 42%)
2022-11-23T02:47:55.7314587Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020716.xml (deflated 43%)
2022-11-23T02:47:55.7315366Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020719.xml (deflated 42%)
2022-11-23T02:47:55.7316196Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020721.xml (deflated 42%)
2022-11-23T02:47:55.7316977Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020723.xml (deflated 41%)
2022-11-23T02:47:55.7317759Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020732.xml (deflated 41%)
2022-11-23T02:47:55.7318533Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020734.xml (deflated 40%)
2022-11-23T02:47:55.7319312Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020743.xml (deflated 41%)
2022-11-23T02:47:55.7320079Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020751.xml (deflated 40%)
2022-11-23T02:47:55.7320864Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020804.xml (deflated 40%)
2022-11-23T02:47:55.7321640Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221123020859.xml (deflated 63%)
2022-11-23T02:47:55.7322462Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20221123020903.xml (deflated 71%)
2022-11-23T02:47:55.7323253Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20221123020907.xml (deflated 43%)
2022-11-23T02:47:55.7324002Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221123020907.xml (deflated 69%)
2022-11-23T02:47:55.7324794Z   adding: test/test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20221123020911.xml (deflated 71%)
2022-11-23T02:47:55.7325657Z   adding: test/test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20221123020911.xml (deflated 40%)
2022-11-23T02:47:55.7326446Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221123020915.xml (deflated 62%)
2022-11-23T02:47:55.7327204Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221123020915.xml (deflated 69%)
2022-11-23T02:47:55.7327950Z   adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20221123020921.xml (deflated 40%)
2022-11-23T02:47:55.7328763Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20221123020928.xml (deflated 71%)
2022-11-23T02:47:55.7329622Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123020935.xml (deflated 45%)
2022-11-23T02:47:55.7330422Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123020942.xml (deflated 41%)
2022-11-23T02:47:55.7331263Z   adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20221123020950.xml (deflated 59%)
2022-11-23T02:47:55.7332319Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221123020959.xml (deflated 61%)
2022-11-23T02:47:55.7333089Z   adding: test/test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20221123021007.xml (deflated 55%)
2022-11-23T02:47:55.7333916Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123021016.xml (deflated 60%)
2022-11-23T02:47:55.7334778Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221123021027.xml (deflated 54%)
2022-11-23T02:47:55.7335791Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221123021040.xml (deflated 74%)
2022-11-23T02:47:55.7336605Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123021054.xml (deflated 61%)
2022-11-23T02:47:55.7337380Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20221123021108.xml (deflated 67%)
2022-11-23T02:47:55.7338200Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20221123021108.xml (deflated 61%)
2022-11-23T02:47:55.7338971Z   adding: test/test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20221123021125.xml (deflated 75%)
2022-11-23T02:47:55.7339730Z   adding: test/test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221123021141.xml (deflated 68%)
2022-11-23T02:47:55.7340544Z   adding: test/test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20221123021157.xml (deflated 79%)
2022-11-23T02:47:55.7341410Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123021216.xml (deflated 67%)
2022-11-23T02:47:55.7342428Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123021216.xml (deflated 43%)
2022-11-23T02:47:55.7343490Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123021216.xml (deflated 44%)
2022-11-23T02:47:55.7344479Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123021238.xml (deflated 56%)
2022-11-23T02:47:55.7345353Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123021238.xml (deflated 77%)
2022-11-23T02:47:55.7346157Z   adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20221123021303.xml (deflated 76%)
2022-11-23T02:47:55.7346957Z   adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20221123021303.xml (deflated 42%)
2022-11-23T02:47:55.7347755Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123021330.xml (deflated 86%)
2022-11-23T02:47:55.7348566Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20221123021400.xml (deflated 77%)
2022-11-23T02:47:55.7349388Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123021433.xml (deflated 64%)
2022-11-23T02:47:55.7350292Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123021433.xml (deflated 84%)
2022-11-23T02:47:55.7351105Z   adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221123021508.xml (deflated 80%)
2022-11-23T02:47:55.7351826Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20221123021546.xml (deflated 75%)
2022-11-23T02:47:55.7352565Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20221123021546.xml (deflated 74%)
2022-11-23T02:47:55.7353298Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123021626.xml (deflated 77%)
2022-11-23T02:47:55.7354032Z   adding: test/test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20221123021725.xml (deflated 84%)
2022-11-23T02:47:55.7354817Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123021831.xml (deflated 90%)
2022-11-23T02:47:55.7355806Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221123021937.xml (deflated 90%)
2022-11-23T02:47:55.7356810Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221123021937.xml (deflated 73%)
2022-11-23T02:47:55.7357685Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221123022212.xml (deflated 93%)
2022-11-23T02:47:55.7358480Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123022548.xml (deflated 96%)
2022-11-23T02:47:55.7359206Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023229.xml (deflated 38%)
2022-11-23T02:47:55.7359884Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023237.xml (deflated 38%)
2022-11-23T02:47:55.7360538Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023245.xml (deflated 39%)
2022-11-23T02:47:55.7361210Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023251.xml (deflated 38%)
2022-11-23T02:47:55.7361924Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023258.xml (deflated 38%)
2022-11-23T02:47:55.7362605Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023305.xml (deflated 39%)
2022-11-23T02:47:55.7363256Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023311.xml (deflated 39%)
2022-11-23T02:47:55.7363925Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023317.xml (deflated 39%)
2022-11-23T02:47:55.7364596Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023323.xml (deflated 38%)
2022-11-23T02:47:55.7365257Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023330.xml (deflated 37%)
2022-11-23T02:47:55.7365903Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023338.xml (deflated 37%)
2022-11-23T02:47:55.7366568Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023345.xml (deflated 37%)
2022-11-23T02:47:55.7367229Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023351.xml (deflated 38%)
2022-11-23T02:47:55.7367889Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023359.xml (deflated 38%)
2022-11-23T02:47:55.7368534Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023406.xml (deflated 39%)
2022-11-23T02:47:55.7369198Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023413.xml (deflated 38%)
2022-11-23T02:47:55.7369856Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023420.xml (deflated 38%)
2022-11-23T02:47:55.7370518Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023427.xml (deflated 37%)
2022-11-23T02:47:55.7371168Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123023435.xml (deflated 37%)
2022-11-23T02:47:55.7372059Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023442.xml (deflated 38%)
2022-11-23T02:47:55.7372761Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023450.xml (deflated 38%)
2022-11-23T02:47:55.7373452Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023457.xml (deflated 38%)
2022-11-23T02:47:55.7374232Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023505.xml (deflated 38%)
2022-11-23T02:47:55.7374929Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023513.xml (deflated 39%)
2022-11-23T02:47:55.7375618Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023520.xml (deflated 38%)
2022-11-23T02:47:55.7376309Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023528.xml (deflated 38%)
2022-11-23T02:47:55.7377047Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023536.xml (deflated 41%)
2022-11-23T02:47:55.7377873Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023544.xml (deflated 41%)
2022-11-23T02:47:55.7378685Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023552.xml (deflated 41%)
2022-11-23T02:47:55.7379509Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023600.xml (deflated 41%)
2022-11-23T02:47:55.7380307Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023608.xml (deflated 41%)
2022-11-23T02:47:55.7381176Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023616.xml (deflated 42%)
2022-11-23T02:47:55.7382004Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023624.xml (deflated 41%)
2022-11-23T02:47:55.7382817Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023632.xml (deflated 41%)
2022-11-23T02:47:55.7383607Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023639.xml (deflated 41%)
2022-11-23T02:47:55.7384410Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023647.xml (deflated 45%)
2022-11-23T02:47:55.7385213Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023655.xml (deflated 45%)
2022-11-23T02:47:55.7386019Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023702.xml (deflated 43%)
2022-11-23T02:47:55.7386807Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023710.xml (deflated 43%)
2022-11-23T02:47:55.7387598Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023718.xml (deflated 45%)
2022-11-23T02:47:55.7388402Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023726.xml (deflated 46%)
2022-11-23T02:47:55.7389208Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023734.xml (deflated 46%)
2022-11-23T02:47:55.7389994Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023741.xml (deflated 47%)
2022-11-23T02:47:55.7390800Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023749.xml (deflated 44%)
2022-11-23T02:47:55.7391598Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023757.xml (deflated 46%)
2022-11-23T02:47:55.7392396Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023805.xml (deflated 45%)
2022-11-23T02:47:55.7393179Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023812.xml (deflated 44%)
2022-11-23T02:47:55.7394050Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023820.xml (deflated 44%)
2022-11-23T02:47:55.7394846Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023828.xml (deflated 42%)
2022-11-23T02:47:55.7395648Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023836.xml (deflated 42%)
2022-11-23T02:47:55.7396432Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023843.xml (deflated 42%)
2022-11-23T02:47:55.7397235Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023851.xml (deflated 44%)
2022-11-23T02:47:55.7398027Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023859.xml (deflated 44%)
2022-11-23T02:47:55.7398829Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023906.xml (deflated 41%)
2022-11-23T02:47:55.7399611Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023913.xml (deflated 44%)
2022-11-23T02:47:55.7400462Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023921.xml (deflated 41%)
2022-11-23T02:47:55.7401277Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023929.xml (deflated 41%)
2022-11-23T02:47:55.7402073Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023936.xml (deflated 41%)
2022-11-23T02:47:55.7402870Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023943.xml (deflated 42%)
2022-11-23T02:47:55.7403662Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023951.xml (deflated 42%)
2022-11-23T02:47:55.7404454Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023959.xml (deflated 42%)
2022-11-23T02:47:55.7405256Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024003.xml (deflated 42%)
2022-11-23T02:47:55.7406056Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024007.xml (deflated 42%)
2022-11-23T02:47:55.7406839Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024011.xml (deflated 42%)
2022-11-23T02:47:55.7407629Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024015.xml (deflated 42%)
2022-11-23T02:47:55.7408430Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024019.xml (deflated 42%)
2022-11-23T02:47:55.7409225Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024023.xml (deflated 42%)
2022-11-23T02:47:55.7410008Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024031.xml (deflated 41%)
2022-11-23T02:47:55.7410805Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024039.xml (deflated 42%)
2022-11-23T02:47:55.7411999Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024047.xml (deflated 41%)
2022-11-23T02:47:55.7412910Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024055.xml (deflated 41%)
2022-11-23T02:47:55.7413699Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024104.xml (deflated 42%)
2022-11-23T02:47:55.7414638Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024110.xml (deflated 41%)
2022-11-23T02:47:55.7415439Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024116.xml (deflated 42%)
2022-11-23T02:47:55.7416245Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024125.xml (deflated 42%)
2022-11-23T02:47:55.7417029Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024133.xml (deflated 41%)
2022-11-23T02:47:55.7417824Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024141.xml (deflated 42%)
2022-11-23T02:47:55.7418624Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024149.xml (deflated 42%)
2022-11-23T02:47:55.7419426Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024155.xml (deflated 42%)
2022-11-23T02:47:55.7420210Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024202.xml (deflated 42%)
2022-11-23T02:47:55.7421068Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024209.xml (deflated 42%)
2022-11-23T02:47:55.7421889Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024215.xml (deflated 43%)
2022-11-23T02:47:55.7422688Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024223.xml (deflated 42%)
2022-11-23T02:47:55.7423471Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024231.xml (deflated 43%)
2022-11-23T02:47:55.7424274Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024255.xml (deflated 44%)
2022-11-23T02:47:55.7425064Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024303.xml (deflated 42%)
2022-11-23T02:47:55.7425865Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024311.xml (deflated 42%)
2022-11-23T02:47:55.7426650Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024317.xml (deflated 41%)
2022-11-23T02:47:55.7427449Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024325.xml (deflated 40%)
2022-11-23T02:47:55.7428240Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024333.xml (deflated 41%)
2022-11-23T02:47:55.7429100Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024341.xml (deflated 41%)
2022-11-23T02:47:55.7429873Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024349.xml (deflated 41%)
2022-11-23T02:47:55.7430651Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024356.xml (deflated 41%)
2022-11-23T02:47:55.7431417Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024402.xml (deflated 42%)
2022-11-23T02:47:55.7432180Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024404.xml (deflated 42%)
2022-11-23T02:47:55.7432921Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024411.xml (deflated 41%)
2022-11-23T02:47:55.7433740Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024417.xml (deflated 41%)
2022-11-23T02:47:55.7434502Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024423.xml (deflated 41%)
2022-11-23T02:47:55.7435267Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024430.xml (deflated 42%)
2022-11-23T02:47:55.7436009Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123024432.xml (deflated 41%)
2022-11-23T02:47:55.7436893Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024438.xml (deflated 42%)
2022-11-23T02:47:55.7437881Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024445.xml (deflated 42%)
2022-11-23T02:47:55.7438858Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024452.xml (deflated 43%)
2022-11-23T02:47:55.7439831Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024459.xml (deflated 42%)
2022-11-23T02:47:55.7440755Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123024506.xml (deflated 41%)
2022-11-23T02:47:55.7441559Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024508.xml (deflated 38%)
2022-11-23T02:47:55.7442322Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024515.xml (deflated 39%)
2022-11-23T02:47:55.7443081Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024523.xml (deflated 39%)
2022-11-23T02:47:55.7443828Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024531.xml (deflated 39%)
2022-11-23T02:47:55.7444587Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024538.xml (deflated 38%)
2022-11-23T02:47:55.7445343Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024546.xml (deflated 39%)
2022-11-23T02:47:55.7446102Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024554.xml (deflated 39%)
2022-11-23T02:47:55.7446843Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024601.xml (deflated 38%)
2022-11-23T02:47:55.7447597Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024609.xml (deflated 39%)
2022-11-23T02:47:55.7448349Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024616.xml (deflated 38%)
2022-11-23T02:47:55.7449101Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024628.xml (deflated 39%)
2022-11-23T02:47:55.7449837Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024635.xml (deflated 39%)
2022-11-23T02:47:55.7450592Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024643.xml (deflated 39%)
2022-11-23T02:47:55.7451345Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024650.xml (deflated 39%)
2022-11-23T02:47:55.7452314Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024658.xml (deflated 39%)
2022-11-23T02:47:55.7453054Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024705.xml (deflated 38%)
2022-11-23T02:47:55.7453903Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024713.xml (deflated 39%)
2022-11-23T02:47:55.7454654Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024720.xml (deflated 39%)
2022-11-23T02:47:55.7455407Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024733.xml (deflated 39%)
2022-11-23T02:47:55.7456135Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123024740.xml (deflated 40%)
2022-11-23T02:47:55.7456854Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123024744.xml (deflated 40%)
2022-11-23T02:47:55.7477462Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T02:47:55.7477866Z # Remove any previous test reports if they exist
2022-11-23T02:47:55.7478196Z rm -f usage-log-*.zip
2022-11-23T02:47:55.7478579Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-11-23T02:47:55.7478981Z # so check to see if the file exists first
2022-11-23T02:47:55.7479277Z if [ -f 'usage_log.txt' ]; then
2022-11-23T02:47:55.7479618Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-11-23T02:47:55.7479919Z fi
2022-11-23T02:47:55.7492209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:47:55.7492532Z env:
2022-11-23T02:47:55.7492778Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:55.7493034Z   GPU_FLAG: --gpus all
2022-11-23T02:47:55.7493403Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:55.7493880Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165
2022-11-23T02:47:55.7494222Z ##[endgroup]
2022-11-23T02:47:55.8257463Z   adding: usage_log.txt (deflated 95%)
2022-11-23T02:47:55.8304185Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T02:47:55.8304480Z with:
2022-11-23T02:47:55.8304748Z   s3-prefix: pytorch/pytorch/3528293554/1/artifact

2022-11-23T02:47:55.8305051Z   retention-days: 14
2022-11-23T02:47:55.8305324Z   if-no-files-found: warn
2022-11-23T02:47:55.8305583Z   path: test-jsons-*.zip
2022-11-23T02:47:55.8305845Z   name: artifact
2022-11-23T02:47:55.8306103Z   s3-bucket: gha-artifacts
2022-11-23T02:47:55.8306360Z   region: us-east-1
2022-11-23T02:47:55.8306595Z env:
2022-11-23T02:47:55.8306841Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:55.8307092Z   GPU_FLAG: --gpus all
2022-11-23T02:47:55.8307460Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:55.8307821Z ##[endgroup]
2022-11-23T02:47:56.2744893Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T02:47:56.2745649Z With the provided path, there will be 1 file uploaded
2022-11-23T02:47:56.2774218Z Uploading to s3 prefix: pytorch/pytorch/3528293554/1/artifact
2022-11-23T02:47:56.2775594Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165.zip
2022-11-23T02:47:56.3945193Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165.zip
2022-11-23T02:47:56.4097224Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T02:47:56.4097501Z with:
2022-11-23T02:47:56.4097784Z   s3-prefix: pytorch/pytorch/3528293554/1/artifact

2022-11-23T02:47:56.4098100Z   retention-days: 14
2022-11-23T02:47:56.4098359Z   if-no-files-found: error
2022-11-23T02:47:56.4098645Z   path: test-reports-*.zip
2022-11-23T02:47:56.4098910Z   name: artifact
2022-11-23T02:47:56.4099147Z   s3-bucket: gha-artifacts
2022-11-23T02:47:56.4099409Z   region: us-east-1
2022-11-23T02:47:56.4099641Z env:
2022-11-23T02:47:56.4099861Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:56.4100129Z   GPU_FLAG: --gpus all
2022-11-23T02:47:56.4100507Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:56.4100999Z ##[endgroup]
2022-11-23T02:47:56.8536134Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T02:47:56.8536872Z With the provided path, there will be 1 file uploaded
2022-11-23T02:47:56.8537259Z Uploading to s3 prefix: pytorch/pytorch/3528293554/1/artifact
2022-11-23T02:47:56.8548277Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165.zip
2022-11-23T02:47:57.0636845Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165.zip
2022-11-23T02:47:57.0793647Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T02:47:57.0793944Z with:
2022-11-23T02:47:57.0794226Z   s3-prefix: pytorch/pytorch/3528293554/1/artifact

2022-11-23T02:47:57.0794522Z   retention-days: 14
2022-11-23T02:47:57.0794777Z   if-no-files-found: ignore
2022-11-23T02:47:57.0795055Z   path: usage-log-*.zip
2022-11-23T02:47:57.0795308Z   name: artifact
2022-11-23T02:47:57.0795542Z   s3-bucket: gha-artifacts
2022-11-23T02:47:57.0795817Z   region: us-east-1
2022-11-23T02:47:57.0796049Z env:
2022-11-23T02:47:57.0796272Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:57.0796538Z   GPU_FLAG: --gpus all
2022-11-23T02:47:57.0796902Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:57.0797234Z ##[endgroup]
2022-11-23T02:47:57.5201411Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T02:47:57.5202280Z With the provided path, there will be 1 file uploaded
2022-11-23T02:47:57.5202912Z Uploading to s3 prefix: pytorch/pytorch/3528293554/1/artifact
2022-11-23T02:47:57.5214130Z Starting upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165.zip
2022-11-23T02:47:57.6927314Z Finished upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9655199165.zip
2022-11-23T02:47:57.7085077Z ##[group]Run # shellcheck disable=SC2156
2022-11-23T02:47:57.7085442Z # shellcheck disable=SC2156
2022-11-23T02:47:57.7085867Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2022-11-23T02:47:57.7099846Z shell: /usr/bin/bash -e {0}
2022-11-23T02:47:57.7100113Z env:
2022-11-23T02:47:57.7100364Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:57.7100622Z   GPU_FLAG: --gpus all
2022-11-23T02:47:57.7100998Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:57.7101364Z ##[endgroup]
2022-11-23T02:47:58.0154769Z ##[group]Run set -x
2022-11-23T02:47:58.0155097Z set -x
2022-11-23T02:47:58.0155411Z python3 -m pip install -r requirements.txt
2022-11-23T02:47:58.0155767Z python3 -m pip install boto3==1.19.12
2022-11-23T02:47:58.0156177Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T02:47:58.0169313Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:47:58.0169620Z env:
2022-11-23T02:47:58.0169869Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:47:58.0170136Z   GPU_FLAG: --gpus all
2022-11-23T02:47:58.0170506Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:47:58.0170884Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T02:47:58.0171130Z   BRANCH: master
2022-11-23T02:47:58.0171389Z   TEST_CONFIG: distributed
2022-11-23T02:47:58.0171910Z   SHARD_NUMBER: 2
2022-11-23T02:47:58.0172242Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T02:47:58.0172589Z   PR_NUMBER: 
2022-11-23T02:47:58.0172834Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T02:47:58.0173126Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T02:47:58.0173438Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T02:47:58.0173695Z   TAG: 
2022-11-23T02:47:58.0173931Z   WORKFLOW_ID: 3528293554
2022-11-23T02:47:58.0174358Z   GITHUB_TOKEN: ***
2022-11-23T02:47:58.0174607Z   GHA_WORKFLOW_JOB_ID: 9655199165
2022-11-23T02:47:58.0174871Z ##[endgroup]
2022-11-23T02:47:58.0204503Z + python3 -m pip install -r requirements.txt
2022-11-23T02:47:58.3165090Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:47:58.3528971Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2022-11-23T02:47:58.3567940Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4)
2022-11-23T02:47:58.3579770Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2022-11-23T02:47:58.3592774Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.58.0)
2022-11-23T02:47:58.4140736Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2022-11-23T02:47:58.4152980Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-11-23T02:47:58.4264737Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2022-11-23T02:47:58.4275912Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-11-23T02:47:58.4542728Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-11-23T02:47:58.4790366Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2022-11-23T02:47:58.4802512Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2022-11-23T02:47:58.4810772Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0)
2022-11-23T02:47:58.4825530Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2022-11-23T02:47:58.4851306Z Requirement already satisfied: filelock in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (3.8.0)
2022-11-23T02:47:58.4955917Z Requirement already satisfied: networkx in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (2.6.3)
2022-11-23T02:47:58.5191395Z Requirement already satisfied: jinja2 in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (3.1.2)
2022-11-23T02:47:58.5224789Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4)
2022-11-23T02:47:58.5249643Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.1.0)
2022-11-23T02:47:58.5618548Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2022-11-23T02:47:58.5632970Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.0.4)
2022-11-23T02:47:58.5656834Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-11-23T02:47:58.5686100Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-11-23T02:47:58.5702342Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12)
2022-11-23T02:47:58.5925662Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-11-23T02:47:58.5938179Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2022-11-23T02:47:58.6019056Z Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from jinja2->-r requirements.txt (line 17)) (2.1.1)
2022-11-23T02:47:58.6731345Z + python3 -m pip install boto3==1.19.12
2022-11-23T02:47:58.9661801Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:47:58.9894209Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2022-11-23T02:47:58.9965933Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2022-11-23T02:47:59.0029229Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2022-11-23T02:47:59.0058168Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2022-11-23T02:47:59.0097949Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12)
2022-11-23T02:47:59.0328894Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2022-11-23T02:47:59.0356755Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-11-23T02:47:59.2890143Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T02:48:15.0994181Z [scribe] Scribe access token not provided, sending report via boto3...
2022-11-23T02:48:15.0994496Z 
2022-11-23T02:48:15.0994853Z ----- Historic stats comparison result ------
2022-11-23T02:48:15.0995094Z 
2022-11-23T02:48:15.0995507Z     job: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T02:48:15.0996338Z     commit: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T02:48:15.0996568Z 
2022-11-23T02:48:15.0996785Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-11-23T02:48:15.0997039Z 
2022-11-23T02:48:15.0997166Z     : (master)
2022-11-23T02:48:15.0997394Z     |
2022-11-23T02:48:15.0997649Z     * 1cfd3858ac (HEAD)              total time  2722.68s
2022-11-23T02:48:15.1000862Z     * 26322544b8 (base)  13 reports, total time  4193.12s ± 2023.52s
2022-11-23T02:48:15.1001360Z     * 7f4b4d2827         13 reports, total time  4157.15s ± 1970.21s
2022-11-23T02:48:15.1001839Z     * b50699f247         13 reports, total time  4190.33s ± 2008.01s
2022-11-23T02:48:15.1002271Z     * 8bf8e4d71e         13 reports, total time  4176.50s ± 2000.30s
2022-11-23T02:48:15.1002702Z     * ce856cee7e         12 reports, total time  3997.88s ± 1945.32s
2022-11-23T02:48:15.1003129Z     * 391b593ca2         13 reports, total time  4190.46s ± 2006.12s
2022-11-23T02:48:15.1003544Z     * 5bba783d21         13 reports, total time  4212.68s ± 2027.67s
2022-11-23T02:48:15.1003970Z     * ea920a1115         13 reports, total time  4210.45s ± 1980.15s
2022-11-23T02:48:15.1004397Z     * 74e62a1fef         13 reports, total time  4199.06s ± 2006.95s
2022-11-23T02:48:15.1004798Z     * 00b7d8ef23         13 reports, total time  4196.42s ± 2038.18s
2022-11-23T02:48:15.1005075Z     |
2022-11-23T02:48:15.1005285Z     :
2022-11-23T02:48:15.1005421Z 
2022-11-23T02:48:15.1005575Z Removed  (across  927 suites)     0 tests, totaling     0.00s
2022-11-23T02:48:15.1005931Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-11-23T02:48:15.1006461Z Added    (across   56 suites)   821 tests, totaling +3469.19s
2022-11-23T02:48:15.1603741Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2022-11-23T02:48:15.1604104Z with:
2022-11-23T02:48:15.1604324Z env:
2022-11-23T02:48:15.1604553Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:48:15.1604829Z   GPU_FLAG: --gpus all
2022-11-23T02:48:15.1605201Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:48:15.1605538Z ##[endgroup]
2022-11-23T02:48:15.1623237Z ##[group]Run set -eou pipefail
2022-11-23T02:48:15.1623546Z set -eou pipefail
2022-11-23T02:48:15.1623803Z 
2022-11-23T02:48:15.1624111Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-11-23T02:48:15.1624456Z for _ in $(seq 1440); do
2022-11-23T02:48:15.1624762Z     # Break if no ssh session exists anymore
2022-11-23T02:48:15.1625050Z     if [ "$(who)" = "" ]; then
2022-11-23T02:48:15.1625313Z       break
2022-11-23T02:48:15.1625586Z     fi
2022-11-23T02:48:15.1625808Z     echo "."
2022-11-23T02:48:15.1626050Z     sleep 5
2022-11-23T02:48:15.1626288Z done
2022-11-23T02:48:15.1639504Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:48:15.1639808Z env:
2022-11-23T02:48:15.1640054Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:48:15.1640305Z   GPU_FLAG: --gpus all
2022-11-23T02:48:15.1640677Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:48:15.1641036Z ##[endgroup]
2022-11-23T02:48:15.1671001Z Holding runner for 2 hours until all ssh sessions have logged out
2022-11-23T02:48:15.1718622Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T02:48:15.1719056Z # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T02:48:15.1719409Z # shellcheck disable=SC2046
2022-11-23T02:48:15.1719712Z docker stop $(docker ps -q) || true
2022-11-23T02:48:15.1720039Z # Prune all of the docker images
2022-11-23T02:48:15.1720344Z docker system prune -af
2022-11-23T02:48:15.1732849Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:48:15.1733161Z env:
2022-11-23T02:48:15.1733411Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:48:15.1733666Z   GPU_FLAG: --gpus all
2022-11-23T02:48:15.1734043Z   DOCKER_CONTAINER_ID: 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:48:15.1734399Z ##[endgroup]
2022-11-23T02:48:15.5736263Z 9de1665d20b3
2022-11-23T02:48:16.6641378Z Deleted Containers:
2022-11-23T02:48:16.6642206Z 9de1665d20b30360c0d9fc079172511cf6bde7f44ad5ad3e1c4b4cb75daf5f5c
2022-11-23T02:48:16.6642566Z 
2022-11-23T02:48:21.6670017Z Deleted Images:
2022-11-23T02:48:21.6671628Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T02:48:21.6673489Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7@sha256:a44ece0129de4f14f08fcb1423a34c97f2f88e2969bfc5ad6f33b15b4dfcfea3
2022-11-23T02:48:21.6674146Z deleted: sha256:ca0e26c7bac33dbcc7a8dbe19f76edee4b91691e765d908dd755388b14acb91e
2022-11-23T02:48:21.6674622Z deleted: sha256:acbff221174f85354b2aba183449be48b8a649044c4d855f47ffe7c15f2f0593
2022-11-23T02:48:21.6675078Z deleted: sha256:6e2b2e86bf8d35dfec9e9487ce66c8ec4c5e366546510dbf2061107259075ec2
2022-11-23T02:48:21.6675535Z deleted: sha256:7ee6329a87fae79992a26743847bbc86bacc2ca03801efcdce7754a05ccb441f
2022-11-23T02:48:21.6675956Z deleted: sha256:6b5c166d5b90baf854740fd230f7d138850141e8dd1069c75900aa09e5e0dc90
2022-11-23T02:48:21.6676410Z deleted: sha256:ab05e7d25d0b30fc6f3cbf720ae9408ac11bcbdfea59731ae72dcc8aa3e520d6
2022-11-23T02:48:21.6676898Z deleted: sha256:f6246cb9656221ce7e6751e93b05e13424fb2583c6137d4edbfef96ceb0fbf4f
2022-11-23T02:48:21.6677598Z deleted: sha256:04d30f9a047eb1f0199adf2b2ca2512f9554b7caad28a2fb14a22a388d305a98
2022-11-23T02:48:21.6678127Z deleted: sha256:5c6550a005fd1407aa8a10de1057e8212385b7ef81fdb403eac9cb25b2cdd30a
2022-11-23T02:48:21.6678574Z deleted: sha256:2db874a48ccf9cf10f3600e1e7e0c5165c462d689f394d7be669d852c0f28f8e
2022-11-23T02:48:21.6679015Z deleted: sha256:1cb9632811ae406a7896c8ad49717c265bba77d94df3a67c23bd0cc1222c93fe
2022-11-23T02:48:21.6679421Z deleted: sha256:0b6a063f7a058d0226b7f958b16f7f2e4b68f39e6275890c7a7a0d339ebf9a70
2022-11-23T02:48:21.6679850Z deleted: sha256:afef4e32922b9a5b37e91ef12e33891a7d22f056ccb366412c89d8b775231c67
2022-11-23T02:48:21.6680303Z deleted: sha256:13dcda7fb091c2ea82bde2883af19edb94358834314a6d7d89de972fddf854cc
2022-11-23T02:48:21.6680766Z deleted: sha256:ba4023bbeb7ba3bdfc8353b5e973c00f1dae850fe0e4232344bca0d7bdc5ed21
2022-11-23T02:48:21.6681192Z deleted: sha256:eda87a7e3963049f09510a9664306ce96a1e843bd2ed9306aad33978f2415934
2022-11-23T02:48:21.6681621Z deleted: sha256:ad917be56ac1ba848d966f7332b79ef401b9420b7952fba45570faf98875de2c
2022-11-23T02:48:21.6682065Z deleted: sha256:3c2ba68cc2c00d49222172449200284f1caf0e9b25d7eeba6698297ef42c7059
2022-11-23T02:48:21.6682480Z deleted: sha256:1505992ac44d07e9564edf19e4ed03a31f7609cbfd0cc4bd607a988bd64d6592
2022-11-23T02:48:21.6682916Z deleted: sha256:7e3ce05cb4796e4f977d34b2c9499faabd362b94750f42ea36023785fd07210b
2022-11-23T02:48:21.6683347Z deleted: sha256:99012b24a24f7cbe1787c76d263d7cf9556d2542c33e1f4fde47d5a9bfced7bf
2022-11-23T02:48:21.6683766Z deleted: sha256:44a93e00ff78481f1922fd26545eba58d27c25ede11210755a18ac1089d34491
2022-11-23T02:48:21.6684158Z deleted: sha256:3d35c57c87aa053605d9839c481d2505ef05d74269206229cd53cf56291a84bc
2022-11-23T02:48:21.6684583Z deleted: sha256:d0644c40eee52273b45cfd6a9d443606fd3b1d570bba60df414bef688531865f
2022-11-23T02:48:21.6685032Z deleted: sha256:87f1587cf0fc998c2fd781ff08bd41ecd3f1dce58e89485eae3f97cf5e65a224
2022-11-23T02:48:21.6685460Z deleted: sha256:e967cbd5936283cfefeee219340c1dc42b04c590734c01da262f3591426d7033
2022-11-23T02:48:21.6685904Z deleted: sha256:cb4e3fb37d95ff54c5467335caa5a165d6951a524c72098c8aa50e91489a1cf7
2022-11-23T02:48:21.6686346Z deleted: sha256:c999a26e008b2a5a646e886f297ad5b2efd1ffc6de561d309eecf7e904d80411
2022-11-23T02:48:21.6686798Z deleted: sha256:0f7c27db48adc87df4d2d2275a157eb1b46971b7b9901cf97c7e5f95a0bd9f3a
2022-11-23T02:48:21.6687219Z deleted: sha256:6172bd74910d9d7ff816aac3ed295122ccc70f926d75b17196d3a8a7da581a48
2022-11-23T02:48:21.6687665Z deleted: sha256:f6c04aacc51d4155e8c2e978c915ad2c1a0d170bf3e933aeeaa66f7f5e9f529b
2022-11-23T02:48:21.6688107Z deleted: sha256:1a123d1e783cb340d39fb0fd6af62f0f31ec366902386b4843d78aa15341ca35
2022-11-23T02:48:21.6688513Z deleted: sha256:14682bd215f05536bf4a50be70610b9092a6f0c799b9adfa102ff191b669013f
2022-11-23T02:48:21.6688931Z deleted: sha256:055f5d43f9fee2ec66f09539d8c1624084e8029d0e74981a53354b7ff44c0af1
2022-11-23T02:48:21.6689364Z deleted: sha256:e1d86ed5bd48d6e66387d31531db825bf43e642ec87d3aea0ae602bda9362204
2022-11-23T02:48:21.6689808Z deleted: sha256:717c3163e25327a177ddf78de4dcdea20521942a9cb18c8e1b21e8336708455b
2022-11-23T02:48:21.6690229Z deleted: sha256:eb3a046c873cc3023f10cc7a3da759dda1fc3977511c26c8fe12d02cd35d9d3b
2022-11-23T02:48:21.6690659Z deleted: sha256:f7d654b861e155438fb4017f993e78f94e4ab971ba70a9923b9578da6f521079
2022-11-23T02:48:21.6691080Z deleted: sha256:9f6e64fe4d84e561aa5f4fb6ca895b16073451674b2e9234678faa71885b7c1a
2022-11-23T02:48:21.6691850Z deleted: sha256:cd4ed903887c2663e7ae16a705cdb277bfac87a35e8b3930e0f7c28f392243ff
2022-11-23T02:48:21.6692316Z deleted: sha256:6acd45a2d76545be71b8be690cb8bf0530cd1322d5aeb90ccd0025b76aeeb9be
2022-11-23T02:48:21.6692752Z deleted: sha256:5f6fad27350d1485f97817591e1e38f341c4f2466973c5afef012cde54640119
2022-11-23T02:48:21.6693151Z deleted: sha256:e77f111f1f3e2f93994c6b428a64619af7dad3012f71096de895d7965c6f6f54
2022-11-23T02:48:21.6693594Z deleted: sha256:1f60aaab75e1b24ad8a6dda4c7591a4cbe1c0233111faa95bbb121e71744f5e2
2022-11-23T02:48:21.6694042Z deleted: sha256:ed3557da864e7a48d82ca4fd51c078d177841f99d52a8480a8442fc6da95b3ef
2022-11-23T02:48:21.6694700Z deleted: sha256:e1cb5ac03c90992794bbcece5ef3a88b8d3dbfebb070b7e7b7dc6e1066a0bd0b
2022-11-23T02:48:21.6695140Z deleted: sha256:92b610b902f3255c897a349782a959cd391014cb7949989ae136548de42631f8
2022-11-23T02:48:21.6695561Z deleted: sha256:c52692915b570d66df58947dd8d7cba68fcfcf8fc296a447481578eb25879692
2022-11-23T02:48:21.6696002Z deleted: sha256:8d99114ec4c3615c49f00fa0fa40e0db8d9445f641c42ffdc127590c6fa9fe9a
2022-11-23T02:48:21.6696426Z deleted: sha256:b3b2969d5ff622447527c54efa6f9d15f8bc70de5db17f46ae11efd709999fbd
2022-11-23T02:48:21.6696876Z deleted: sha256:bfb13edd0300dc2515d887adb670832f4a85848e65db3348bbb1fd74e90fc5fc
2022-11-23T02:48:21.6697328Z deleted: sha256:c1ca91bea3c8e97f0d48fbfe60f987231f95874fcf29a94b76bb67dca54cad18
2022-11-23T02:48:21.6697772Z deleted: sha256:b5ef1223e23564cc06af079be55d6a3962040fb669c5d8724912ff6ea2075e59
2022-11-23T02:48:21.6698182Z deleted: sha256:69f57fbceb1b420d7e4697e0f6514887b0805ee0059bea7d51e0a832962e74bf
2022-11-23T02:48:21.6698432Z 
2022-11-23T02:48:21.6698580Z Total reclaimed space: 20.08GB
2022-11-23T02:48:21.6747681Z Post job cleanup.
2022-11-23T02:48:21.6785608Z Post job cleanup.
2022-11-23T02:48:21.8203733Z [command]/usr/bin/git version
2022-11-23T02:48:21.8267815Z git version 2.37.1
2022-11-23T02:48:21.8335308Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/cbfcdc0e-67ba-43e0-8ed1-5bcc5236dfc6' before making global git config changes
2022-11-23T02:48:21.8336336Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T02:48:21.8342168Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T02:48:21.8386896Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T02:48:21.8425308Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T02:48:21.8752951Z Entering 'android/libs/fbjni'
2022-11-23T02:48:21.8796381Z Entering 'third_party/FP16'
2022-11-23T02:48:21.8839047Z Entering 'third_party/FXdiv'
2022-11-23T02:48:21.8882515Z Entering 'third_party/NNPACK'
2022-11-23T02:48:21.8926095Z Entering 'third_party/QNNPACK'
2022-11-23T02:48:21.8968924Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T02:48:21.9012217Z Entering 'third_party/XNNPACK'
2022-11-23T02:48:21.9067774Z Entering 'third_party/benchmark'
2022-11-23T02:48:21.9110347Z Entering 'third_party/cpuinfo'
2022-11-23T02:48:21.9154076Z Entering 'third_party/cub'
2022-11-23T02:48:21.9197506Z Entering 'third_party/cudnn_frontend'
2022-11-23T02:48:21.9246183Z Entering 'third_party/cutlass'
2022-11-23T02:48:21.9298068Z Entering 'third_party/eigen'
2022-11-23T02:48:21.9343631Z Entering 'third_party/fbgemm'
2022-11-23T02:48:21.9386653Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T02:48:21.9428695Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T02:48:21.9471542Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T02:48:21.9514353Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T02:48:21.9557985Z Entering 'third_party/flatbuffers'
2022-11-23T02:48:21.9602803Z Entering 'third_party/fmt'
2022-11-23T02:48:21.9646253Z Entering 'third_party/foxi'
2022-11-23T02:48:21.9688898Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T02:48:21.9732206Z Entering 'third_party/gloo'
2022-11-23T02:48:21.9775645Z Entering 'third_party/googletest'
2022-11-23T02:48:21.9819234Z Entering 'third_party/ideep'
2022-11-23T02:48:21.9861088Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T02:48:21.9906582Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T02:48:21.9957638Z Entering 'third_party/ios-cmake'
2022-11-23T02:48:22.0000967Z Entering 'third_party/ittapi'
2022-11-23T02:48:22.0044283Z Entering 'third_party/kineto'
2022-11-23T02:48:22.0087303Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T02:48:22.0130930Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T02:48:22.0175516Z Entering 'third_party/nccl/nccl'
2022-11-23T02:48:22.0219977Z Entering 'third_party/neon2sse'
2022-11-23T02:48:22.0263017Z Entering 'third_party/nlohmann'
2022-11-23T02:48:22.0307345Z Entering 'third_party/onnx'
2022-11-23T02:48:22.0364700Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T02:48:22.0407780Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T02:48:22.0455193Z Entering 'third_party/onnx-tensorrt'
2022-11-23T02:48:22.0498387Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T02:48:22.0545613Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T02:48:22.0588812Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T02:48:22.0632318Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T02:48:22.0680376Z Entering 'third_party/pocketfft'
2022-11-23T02:48:22.0723211Z Entering 'third_party/protobuf'
2022-11-23T02:48:22.0771897Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T02:48:22.0814262Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T02:48:22.0858847Z Entering 'third_party/psimd'
2022-11-23T02:48:22.0902432Z Entering 'third_party/pthreadpool'
2022-11-23T02:48:22.0945277Z Entering 'third_party/pybind11'
2022-11-23T02:48:22.0988477Z Entering 'third_party/python-enum'
2022-11-23T02:48:22.1031734Z Entering 'third_party/python-peachpy'
2022-11-23T02:48:22.1074987Z Entering 'third_party/python-six'
2022-11-23T02:48:22.1117626Z Entering 'third_party/sleef'
2022-11-23T02:48:22.1161338Z Entering 'third_party/tbb'
2022-11-23T02:48:22.1207178Z Entering 'third_party/tensorpipe'
2022-11-23T02:48:22.1250991Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T02:48:22.1294719Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T02:48:22.1343245Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T02:48:22.1386291Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T02:48:22.1428176Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T02:48:22.1473042Z Entering 'third_party/zstd'
2022-11-23T02:48:22.1537393Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T02:48:22.1568739Z http.https://github.com/.extraheader
2022-11-23T02:48:22.1579690Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-11-23T02:48:22.1618180Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T02:48:22.1938392Z Entering 'android/libs/fbjni'
2022-11-23T02:48:22.1963126Z http.https://github.com/.extraheader
2022-11-23T02:48:22.1996550Z Entering 'third_party/FP16'
2022-11-23T02:48:22.2021526Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2054394Z Entering 'third_party/FXdiv'
2022-11-23T02:48:22.2078823Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2112475Z Entering 'third_party/NNPACK'
2022-11-23T02:48:22.2137795Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2170220Z Entering 'third_party/QNNPACK'
2022-11-23T02:48:22.2195603Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2228236Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T02:48:22.2252803Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2285809Z Entering 'third_party/XNNPACK'
2022-11-23T02:48:22.2310890Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2355921Z Entering 'third_party/benchmark'
2022-11-23T02:48:22.2380911Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2412968Z Entering 'third_party/cpuinfo'
2022-11-23T02:48:22.2437769Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2470556Z Entering 'third_party/cub'
2022-11-23T02:48:22.2495622Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2528213Z Entering 'third_party/cudnn_frontend'
2022-11-23T02:48:22.2554367Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2592331Z Entering 'third_party/cutlass'
2022-11-23T02:48:22.2617548Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2658505Z Entering 'third_party/eigen'
2022-11-23T02:48:22.2683152Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2717916Z Entering 'third_party/fbgemm'
2022-11-23T02:48:22.2743067Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2776118Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T02:48:22.2799623Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2832123Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T02:48:22.2857533Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2890071Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T02:48:22.2914160Z http.https://github.com/.extraheader
2022-11-23T02:48:22.2946491Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T02:48:22.2969710Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3003343Z Entering 'third_party/flatbuffers'
2022-11-23T02:48:22.3028379Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3063793Z Entering 'third_party/fmt'
2022-11-23T02:48:22.3087790Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3120279Z Entering 'third_party/foxi'
2022-11-23T02:48:22.3144559Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3176615Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T02:48:22.3201532Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3234122Z Entering 'third_party/gloo'
2022-11-23T02:48:22.3259383Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3291233Z Entering 'third_party/googletest'
2022-11-23T02:48:22.3316925Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3349402Z Entering 'third_party/ideep'
2022-11-23T02:48:22.3374062Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3405326Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T02:48:22.3430243Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3465142Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T02:48:22.3489253Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3529556Z Entering 'third_party/ios-cmake'
2022-11-23T02:48:22.3553965Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3585600Z Entering 'third_party/ittapi'
2022-11-23T02:48:22.3610075Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3642127Z Entering 'third_party/kineto'
2022-11-23T02:48:22.3667783Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3699969Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T02:48:22.3725382Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3758787Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T02:48:22.3783008Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3817908Z Entering 'third_party/nccl/nccl'
2022-11-23T02:48:22.3842086Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3874738Z Entering 'third_party/neon2sse'
2022-11-23T02:48:22.3899434Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3930687Z Entering 'third_party/nlohmann'
2022-11-23T02:48:22.3955805Z http.https://github.com/.extraheader
2022-11-23T02:48:22.3990384Z Entering 'third_party/onnx'
2022-11-23T02:48:22.4014939Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4061384Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T02:48:22.4086838Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4120027Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T02:48:22.4145297Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4183229Z Entering 'third_party/onnx-tensorrt'
2022-11-23T02:48:22.4207469Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4241399Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T02:48:22.4266687Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4306379Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T02:48:22.4330208Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4363694Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T02:48:22.4389183Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4423623Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T02:48:22.4447333Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4485188Z Entering 'third_party/pocketfft'
2022-11-23T02:48:22.4510768Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4543763Z Entering 'third_party/protobuf'
2022-11-23T02:48:22.4568704Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4606528Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T02:48:22.4631422Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4665003Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T02:48:22.4688641Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4723948Z Entering 'third_party/psimd'
2022-11-23T02:48:22.4750000Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4783097Z Entering 'third_party/pthreadpool'
2022-11-23T02:48:22.4807713Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4840775Z Entering 'third_party/pybind11'
2022-11-23T02:48:22.4866046Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4899524Z Entering 'third_party/python-enum'
2022-11-23T02:48:22.4924325Z http.https://github.com/.extraheader
2022-11-23T02:48:22.4957215Z Entering 'third_party/python-peachpy'
2022-11-23T02:48:22.4982355Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5015677Z Entering 'third_party/python-six'
2022-11-23T02:48:22.5040319Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5073550Z Entering 'third_party/sleef'
2022-11-23T02:48:22.5098937Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5131946Z Entering 'third_party/tbb'
2022-11-23T02:48:22.5156459Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5191919Z Entering 'third_party/tensorpipe'
2022-11-23T02:48:22.5217409Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5250229Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T02:48:22.5274872Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5308998Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T02:48:22.5333202Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5365544Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T02:48:22.5390665Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5424018Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T02:48:22.5448340Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5481120Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T02:48:22.5505945Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5543399Z Entering 'third_party/zstd'
2022-11-23T02:48:22.5568396Z http.https://github.com/.extraheader
2022-11-23T02:48:22.5881917Z Cleaning up orphan processes